Low variance region detection for improved detection

ABSTRACT

Using detection of low variance regions for improving detection is described. In an example, sensor data can be received from a sensor associated with a vehicle. The sensor data can represent an environment. An indication of a low variance region associated with the sensor data can be determined and an indication of a high variance region associated with the sensor data can be determined based at least in part on the indication of the low variance region. The vehicle can be controlled based on at least one of the sensor data or the indication of the high variance region.

PRIORITY

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/925,698, filed on Oct. 24, 2019, the entire contents of which areincorporated by reference herein. Furthermore, this application claimspriority to, and is a continuation-in-part of U.S. patent applicationSer. No. 16/457,524, filed on Jun. 28, 2019, the entire contents ofwhich are incorporated by reference herein.

BACKGROUND

An autonomous vehicle often includes a perception system to capturesensor data of an environment. The perception system allows theautonomous vehicle to recognize objects in the environment so that theautonomous vehicle can plan a safe route through the environment. Thesafe operation of an autonomous vehicle depends, at least in part, oninformation made available by the perception system in detecting,classifying, and predicting motion of objects. Thus, inaccurate and/orincomplete information provided by such detection, classification,and/or prediction of objects can reduce the safety with which autonomousvehicles can operate.

The perception system can utilize computer vision techniques forrecognizing objects in the environment. Computer vision techniquesenable computers to analyze image data to understand the content of theimage data. Examples of computer vision techniques include objectclassification, object identification, object verification, objectdetection, object landmark detection, object segmentation, objectrecognition, and so on. When solving computer vision problems, there isa significant amount of variance which can affect the accuracy ofcomputer vision solutions. That is, one long standing issue withcomputer vision, and more broadly with the field of deep learning, is tobe able to solve for the long tail of the data distribution. As notedabove, this can be problematic for perception systems onboard autonomousvehicles, as well as for other uses of computer vision techniques.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical components or features.

FIG. 1 illustrates an example process for performing techniquesdescribed herein.

FIG. 2A illustrates details associated with an output of a headdetection process.

FIG. 2B illustrates details associated with an output of a pedestriandetection process.

FIG. 3 is a block diagram illustrating an example system for performingtechniques described herein.

FIG. 4 illustrates an example process for using head detectiontechniques for improving pedestrian detection, as described herein.

FIG. 5 illustrates an example process for determining whether apedestrian is present in image data associated with an environment usingmachine learned models, as described herein.

FIG. 6 illustrates another example process for determining whether apedestrian is present in image data associated with an environment usingmachine learned models, as described herein.

FIG. 7 illustrates yet another example process for determining whether apedestrian is present in image data associated with an environment usingmachine learned models, as described herein.

FIG. 8 illustrates an example process for performing techniquesdescribed herein.

FIG. 9 illustrates an example process for using sub-region techniquesfor improving detection, as described herein.

FIG. 10 illustrates an example process for determining whether a highvariance region is present in image data associated with an environmentusing machine learned models, as described herein.

FIG. 11 illustrates another example process for determining whether ahigh variance region is present in image data associated with anenvironment using machine learned models, as described herein.

FIG. 12 illustrates yet another example process for determining whethera high variance region is present in image data associated with anenvironment using machine learned models, as described herein.

FIG. 13 illustrates an example process for training a model to detecthigh variance region(s) and/or low variance region(s).

DETAILED DESCRIPTION

Techniques described herein relate to using low variance region, orsub-region, detection for improving detection, such as high varianceregion detection. In an example, image data associated with anenvironment within which a vehicle is positioned can be analyzed using amachine learned algorithm to detect a low variance region. A lowvariance region can be a collection of points forming amulti-dimensional object, or portion thereof, wherein variance betweenmultiple instances of the multi-dimensional object is below a threshold.Based at least partly on detecting the low variance region associatedwith the image data, a high variance region can be determined to bepresent in the environment. A high variance region can be a collectionof points forming a multi-dimensional object, or portion thereof,wherein variance between multiple instances of the multi-dimensionalobject meets or exceeds a threshold. That is, techniques describedherein relate to using a sub-region of a high variance region to detectthe high variance region in the image data. A region of interestcorresponding to the high variance region can be identified in the imagedata and a bounding box can be associated with the region of interest inthe image data. That is, the bounding box can be used to identify aposition and/or extent of the high variance region in the environment.The image data, including the bounding box, can be provided to a systemof the vehicle for making determinations with respect to controlling thevehicle. For instance, an indication of the high variance region can beprovided to a planning system or a prediction system associated with thevehicle to control the vehicle within the environment.

In one example, a low variance region that may correspond to a head of aperson. In such an example, a head can be used for improving detectionof a high variance region, which can be a pedestrian. Techniquesdescribed herein are directed to using head detection for improvingpedestrian detection. In an example, image data associated with anenvironment within which a vehicle is positioned can be analyzed using amachine learned algorithm to detect a head, or any portion thereof(e.g., a front view of a face, a side view of a face, a portion of aface, a back view of a head, etc.). Based at least partly on detectingthe head in the image data, a pedestrian can be determined to be presentin the environment. A region of interest corresponding to the pedestriancan be identified in the image data and a bounding box can be associatedwith the region of interest in the image data. That is, the bounding boxcan be used to identify a position and/or extent of the pedestrian inthe environment. The image data, including the bounding box, can beprovided to a system of the vehicle for making determinations withrespect to controlling the vehicle. For instance, an indication of thepedestrian can be provided to a planning system or a prediction systemassociated with the vehicle to control the vehicle within theenvironment.

Additional and alternative examples can be imagined. For example, a lowvariance region can correspond to headlights of a vehicle and detectionof such headlights can be used to improve detection of a high varianceregion, such as a vehicle (e.g., particularly at night or in low light).Furthermore, a low variance region can correspond to a wheel of avehicle and detection of such a wheel can be used to improve detectionof a high variance region, such as a vehicle. As another example, a lowvariance region can correspond to a door of a house and detection ofsuch a door can be used to improve detection of a high variance region,such as a house. In some examples, low variance and high variancecorrespondence can be learned by machine learning techniques. In someexamples, low variance and high variance correspondence can be explicit(e.g., by pre-informing network(s) of such correspondence). In someexamples, machine trained models can be trained to detect low variancefeatures, which can serve as proxies for high variance regions, withoutexplicitly modelling low variance outputs. Where described in someexamples herein as a low variance detection (or region) informing a highvariance detection, such examples are meant to be inclusive of multiplenetworks (e.g., in which a low variance network output is used toprovide additional context to a high variance detector network), singlenetworks trained to determine regions of low variance (e.g., havinglosses associated therewith which are based at least in part on variancein the input region) and couple to regions of general detections (e.g.,by coupling detection losses with variance-based losses), and/or anycombination thereof, etc.

Object detection is a computer vision technique for identifying objects(e.g., pedestrians, cyclists, vehicles, animals, etc.) in image data.Object classification is a computer vision technique for classifyingidentified objects in image data. In some examples, a computing systemonboard a vehicle (e.g., a perception system) can identify and/orclassify objects in image data associated with an environment withinwhich the vehicle is positioned, for example, for determining how tocontrol the vehicle. In at least one example, a computing system onboarda vehicle can identify and/or classify other vehicles, cyclists,pedestrians, and the like in image data associated with an environmentwithin which the vehicle is positioned to determine how the vehicleshould navigate within the environment.

Some objects are associated with significant variance. Variance canrefer to differences in input data. When multiple instances of an object(e.g., objects classified with a same bounding box), or a portionthereof, associated with input data (e.g., image data) are compared, avariance metric associated with the multiple instances of an object, ora portion thereof, that meets or exceeds a threshold, such objects, orportions thereof, can be associated with “high variance regions.” Such avariance metric can represent the variance of information inside of abounding box associated with an object, or portion thereof, in inputdata, batches of input data, or an entire input space. In some examples,the variance metric can be based on entropy, pixel intensity, aspectratios between boxes, and the like. Detecting such high variance regionscan be difficult because of the variance between multiple instances ofsuch regions. That is, modeling high variance regions, for example,using neural networks, can be challenging due to the lack of priors thatlend themselves to region learning. As such, current detection and/orclassification techniques can be inaccurate (e.g., miss a detection orhave a high uncertainty associated therewith). Furthermore, because highvariance regions can be non-convex, modeling such high variance regionsrequires a significant amount of network energy, processing time, and/orother resources. Increased accuracy can be achieved by increasing theamount of data used for training machine learned algorithms to accountfor the variation described above. However, increasing the amount ofdata used for training machine learned algorithms can cause an increasein the amount of training required, and the amount of compute requiredto perform such training.

As an example, detecting and/or classifying pedestrians can be difficultbecause pedestrians can be associated with different heights, widths,etc., as well as lie in a variety of poses in an environment (e.g.,laying down, working underneath a car, on a ladder, etc.). That is,there can be significant variation in pedestrians such that currentdetection and/or classification techniques can be inaccurate (e.g., missa detection or have a high uncertainty associated therewith).Furthermore, variation in lighting, scale, rotation, occlusion, etc. canfurther cause current detection and/or classification techniques to beinaccurate. Increased accuracy can be achieved by increasing the amountof data used for training machine learned algorithms to account for thevariation described above. However, increasing the amount of data usedfor training machine learned algorithms can cause an increase in theamount of training required, and the amount of compute required toperform such training.

As described above, techniques described herein relate to using lowvariance region detection for improving detections generally (which mayinclude, for example, high variance-type detections). Low varianceregions are “low variance” because when multiple instances of an object(e.g., objects classified with a same bounding box), or portion thereof,associated with input data (e.g., image data) are compared, a variancemetric associated with the multiple instances of an object, or portionthereof, can be below a threshold. That is, low variance regions can beassociated with a variance metric associated with the multiple instancesof an object, or portion thereof, in input data, batches of input data,or an entire input space that is below a threshold. In some examples, alow variance region can be a sub-region of a high variance region. As anexample, a region of image data associated with a face is associatedwith a low variance region and a region of image data associated with apedestrian is associated with a high variance region. As anotherexample, a region of image data associated with a door is associatedwith a low variance region and a region of image data associated with ahouse is associated with a high variance region.

Detecting low variance regions can be easier than detecting highvariance regions because the variance between multiple instances of lowvariance regions is lower and thus requires less training and compute.That is, instead of trying to model high variance regions, techniquesdescribed herein attempt to solve the problem in a bottom up manner,wherein a sub-region/low variance region is detected first, and such asub-region/low variance region can be used to detect a high varianceregion. By breaking up the problem (e.g., modeling high varianceregions) into sub-parts, the variance in the sub-parts can be lower thanthe variance of the high variance regions, and the variance of the highvariance regions can be in a disjointed set of parameters.

In an example, image data associated with an environment within which avehicle is positioned can be analyzed using a machine learned algorithmto detect a low variance region. Based at least partly on detecting thelow variance region associated with the image data, a high varianceregion can be determined to be present in the environment. In someexamples, the low variance region can be detected in image data using aneural network. A neural network is a framework of machine learnedmodels that work together to classify inputs (e.g., of image data) basedon a previous training process. In some examples, a neural networkclassifies image data as either containing the low variance region ornot, based upon the model being previously trained on a set of imageshaving instances of the low variance region and not having instances ofthe low variance region.

As described above, the computing system can identify a region ofinterest corresponding to the high variance region in the image dataand, in some examples, can associate a bounding box with the region ofinterest. In at least some examples, the region may be determined fromthe low variance region detection alone (e.g., by region proposals asdescribed in detail herein). In additional or alternate examples, a lowvariance region detection network may be incorporated with one or otherportions of a larger network (e.g., where low variance region detectionis used as a feature input into another network trained to detect highvariance regions), as a single network with different network lowvariance regions trained jointly with the low variance region detectionnetwork in which similar features are shared, and the like. The imagedata, which can include an indication of the region of interestcorresponding to the high variance region (e.g., the bounding box), canbe provided to a system of the computing system that makesdeterminations with respect to controlling the vehicle.

For instance, the image data, including the indication of the highvariance region, can be provided to a planning system or a predictionsystem associated with the computing system to control the vehiclewithin the environment. In some such examples, training may be performedby a user specifying corresponding low variance features compared tohigh variance detections (e.g., training a face detector to inform apedestrian detector). In other examples, such training may be performedby having such a model (e.g., neural network) learn proposed lowvariance features which should be associated with detections based onhigh variance input (e.g., incorporating losses based on inputvariance). In some examples, training may be performed by causing inputimages to be projected into feature spaces and subsequently projectedback into image spaces, whereby a loss function can be used to enforceconsistency between original input images and subsequently reconstructedimages for modelling high variance regions end-to-end as an optimizationproblem. Additional details are described below.

Techniques described herein enable high variance regions to be detectedwith more accuracy than what is available with current techniques. Thatis, by breaking a problem into low variance components and high variancecomponents, machine learned model(s) can provide more certaindeterminations and/or can provide determinations about data faster thanwhat is available with current techniques. That is, instead of usingtraining data that includes significant variations of high varianceregions and/or environmental conditions, techniques described herein canstreamline computer vision techniques by using machine learningtechniques to determine the presence of high variance regions in imagedata. That is, techniques described herein improve computer visiontechniques and/or technologies by increasing the accuracy at whichcomputer vision systems can determine the presence of high varianceregions without requiring increases in training data and computerresources.

As an example, techniques described herein relate to using headdetection for improving pedestrian detection. As used herein, “headdetection,” can refer to the use of facial detection techniques fordetecting heads and/or faces, or portions thereof, in image data. Facialdetection is a computer vision technique for identifying human faces inimage data. This technology has several different uses, which caninclude tagging pictures in social networking sites to biometrics andhealthcare to security access control. In some examples, human faces canbe detected in image data using neural networks. A neural network is aframework of machine learned models that work together to classifyinputs (e.g., of image data) based on a previous training process. Insome examples, a neural network classifies image data as eithercontaining a human face or not, based upon the model being previouslytrained on a set of facial and non-facial images. Examples of facialrecognition models include, but are not limited to, FaceMatch, FaceEverything, Fisherfaces, DeepID Test, Tensorflow, OpenFace, TinyFaces,etc.

In an example, a computing system (e.g., a perception system) onboard avehicle can receive image data of an environment within which thevehicle is positioned. The computing system can analyze the image datausing facial detection techniques described above to identify a head inthe environment. Responsive to detecting a head in the image data, thecomputing system onboard a vehicle can determine that a pedestrian ispresent in the environment. As described above, the computing system canidentify a region of interest corresponding to the pedestrian in theimage data and, in some examples, can associate a bounding box with theregion of interest. In at least some examples, the region may bedetermined from the head detection alone (e.g., by region proposals asdescribed in detail herein). In additional or alternate examples, a headdetection network may be incorporated with one or other portions of alarger network (e.g., where head detection is used as a feature inputinto another network trained to detect pedestrians), as a single networkwith different network heads trained jointly with the head detectionnetwork in which similar features are shared, and the like. The imagedata, which can include an indication of the region of interestcorresponding to the pedestrian (e.g., the bounding box), can beprovided to a system of the computing system that makes determinationswith respect to controlling the vehicle. For instance, the image data,including the indication of the pedestrian, can be provided to aplanning system or a prediction system associated with the computingsystem to control the vehicle within the environment. Additional detailsare described below.

Techniques described herein enable pedestrians to be detected with moreaccuracy than what is available with current techniques. That is, byusing head detection to determine that pedestrians are present in imagedata, techniques described herein can determine pedestrians are presentdespite variations of pedestrian heights, widths, poses (relativepositions and orientations of arms, legs, torso, etc.), and so on.Instead of using training data that includes significant variations ofpedestrians and/or environmental conditions, techniques described hereincan streamline computer vision techniques by using facial recognitiontechniques to determine the presence of pedestrians in image data. Thatis, techniques described herein improve computer vision techniquesand/or technologies by increasing the accuracy at which computer visionsystems can determine the presence of pedestrians without requiringincreases in training data and computer resources.

While techniques described above are directed to the detection ofpedestrians, techniques described herein may be applied to any number oftypes of objects (e.g., animals, cyclists, etc.). That is, techniquesdescribed herein should not be construed as being limited to thedetection of pedestrians using head detection. As described above,techniques described above can be applicable for any regions (e.g.,collections of points forming a multi-dimensional object, or portionthereof) wherein a low variance region can be used to improve detectionof a high variance region.

Furthermore, while techniques described above are directed to detectinglow variance regions and high variance regions in image data, techniquesdescribed herein can be applicable to any type of input data (e.g.,LIDAR data, RADAR data, audio sensor data, thermal sensor data, pressuresensor data, etc.), which may or may not be within the domain ofautonomous vehicles.

The techniques described herein can be implemented in a number of ways.Example implementations are provided below with reference to thefollowing figures. Example implementations are discussed in the contextof autonomous vehicles; however, the methods, apparatuses, and systemsdescribed herein can be applied to a variety of systems (e.g., a sensorsystem or a robotic platform), and are not limited to autonomousvehicles. In one example, the techniques described herein may beutilized in driver-controlled vehicles in which such a system mayprovide an indication to a driver of the vehicle of whether it is safeto perform various maneuvers. In another example, the techniques can beutilized in an aviation or nautical context, or in any system involvingobjects or entities that may be associated with behavior that is unknownto the system. In some examples, the techniques can be utilized incontexts outside of autonomous vehicles and/or perception systems. Thatis, in some examples, techniques described herein can be applicable toweather data, financial data, etc. Furthermore, the techniques describedherein can be used with real data (e.g., captured using sensor(s)),simulated data (e.g., generated by a simulator), or any combination ofthe two.

FIG. 8 illustrates an example process 800 for performing techniquesdescribed herein. In FIG. 8, a first detector 802, which can detect lowvariance regions and can be associated with a perception system asdescribed below, can receive sensor data, such as image data 804. Theimage data 804 can include one or more frames of images of anenvironment within which a vehicle 806 is positioned. In at least oneexample, the first detector 802 (and the perception system) can beonboard the vehicle 806. In at least one example, the vehicle 806 can bean autonomous vehicle configured to operate according to a Level 5classification issued by the U.S. National Highway Traffic SafetyAdministration, which describes a vehicle capable of performing allsafety-critical functions for the entire trip, with the driver (oroccupant) not being expected to control the vehicle at any time. In suchan example, since the vehicle 806 can be configured to control allfunctions from start to stop, including all parking functions, it can beunoccupied. This is merely an example, and the systems and methodsdescribed herein can be incorporated into any ground-borne, airborne, orwaterborne vehicle, including those ranging from vehicles that need tobe manually controlled by a driver at all times, to those that arepartially or fully autonomously controlled. That is, in the illustratedexample, the vehicle 806 is an autonomous vehicle; however, the vehicle806 could be any other type of vehicle.

In at least one example, the first detector 802 can receive the imagedata 804 and can analyze the image data 804 using a machine learnedmodel. As described above, in at least one example, the first detector802 can be part of a computing system onboard the vehicle 806 that canutilize a neural network for detecting low variance region(s) in theimage data 804. However, in additional or alternative examples, thefirst detector 802 can use any other type of machine learned model, asdescribed below with reference to FIG. 3. The first detector 802 cangenerate an output 808 based at least in part on analyzing the imagedata 804 with the machine learned model. In at least one example, theoutput 808 can include the image data 804 with one or more indications880 of whether a low variance region was detected by the first detector802. In some examples, the indication(s) 880 can be bounding box(es)that surround region(s) of interest corresponding to low varianceregion(s) in the image data 804 and/or corresponding certaintyinformation. In other examples, any other type of indication can be usedto indicate whether a low variance region was detected by the firstdetector 802. In at least one example, the indication(s) 810 can beassociated with information including an x-position (global and/or localposition), a y-position (global and/or local position), a z-position(global and/or local position), an orientation (e.g., a roll, pitch,yaw), an extent of the object (e.g., height (h), width (w), etc.), anindication of whether a low variance region was detected (e.g., a binaryindication (i.e., low variance region detected (H) or not (N)), aprobability, etc.), etc. In some examples, the information can be usedto generate the bounding box(es) and/or other indications. Additionaldetails are illustrated in FIG. 2A.

In some examples, the first detector 802 can be associated with amulti-class classifier, which can analyze the image data 804 and outputa plurality of classifications with corresponding certainty information(e.g., probability of a detected object corresponding to a particularclassification). In such an example, a machine learned model can betrained to classify low variance regions into classifications such as alow variance region of a first class, a low variance region of a secondclass, a low variance region of a third class, etc. Such classificationsand/or corresponding certainty information can be useful for determiningindications of low variance regions, as described herein. As describedbelow with reference to FIG. 3, such classifications can be provided tothe networks (e.g., to establish a relationship between the low varianceregion and each class) or the networks can learn the relationshipbetween the low variance region and each class (e.g., using end-to-endtraining).

In at least one example, the output 808 of the first detector 802 can beinput into a second detector 812. The second detector 812 can be trainedto detect regions (e.g., objects) in the image data. In at least oneexample, detection of high variance regions by the second detector 812can be improved based at least in part on the output 808 (e.g.,information associated with the low variance output). In at least oneexample, the second detector 812 can determine that a high varianceregion is present in the environment within which the vehicle 806 ispresented and/or information associated with the high variance region.That is, the second detector 812 can analyze the output 808 and candetermine that a high variance region is present in the environmentwithin which the vehicle 806 is presented and/or information associatedwith the high variance region. In at least one example, the seconddetector 812 can be associated with a perception system of the vehicle806, as described below. In some examples, the second detector 812 canuse one or more statistical models to determine that a high varianceregion is present in an environment (e.g., based on a size and/orposition of the bounding box of the output 808, estimating (based on astatistical likelihood) a proposed region for a high variance region inthe image) (or not). In additional or alternative examples, the seconddetector 812 can use one or more machine learned models, which cananalyze the output 808 and, in some examples, additional data asdescribed below, to determine that a high variance region is present inan environment (or not). In some examples, the image data 804 can beinput directly into the second detector 812 in addition to the firstdetector 802. Additional details are described below with reference toFIGS. 9-12. Regardless, such a second detector 812 may utilizeinformation provided by the first detector 802 in order to better informsuch detections (whether high variance or not). In any such examples,such a detector may be trained independently (e.g., from the firstdetector 802, and, in some examples, pre-informed of relationshipsbetween low variance regions and high variance regions) and/orend-to-end. When trained end-to-end, such training may enforce thecoupling between low variance regions which inform the ultimate output.In some examples, training such a first detector 802 may compriseevaluating and back-propagating based at least in part on a lossfunction which determines a measure of the variance of an associatedregion in the input space.

The second detector 812 can generate an output 814 that is based atleast partly on the output 808. In at least one example, the output 814of the second detector 812 can include one or more indications 816 thatare associated with regions of interest corresponding to identified highvariance region(s) in the environment within which the vehicle 806 ispositioned. In some examples, the indication(s) 816 can be boundingbox(es) that surround region(s) of interest corresponding to highvariance region(s) in the image data 804. In other examples, any othertype of indication can be used to indicate that a high variance regionwas detected in the environment by the second detector 812. In at leastone example, the indication(s) 816 can be associated with informationincluding an x-position (global and/or local position), a y-position(global and/or local position), a z-position (global and/or localposition), an orientation (e.g., a roll, pitch, yaw), an object type(e.g., a classification), a velocity of the object, an acceleration ofthe object, an extent of the object (e.g., height (h), width (w), etc.),etc. In some examples, the information can be used to generate thebounding box(es) and/or other indications. Additional details areillustrated in FIG. 2B. In some such examples, a user may manuallyspecify classes and subclasses for coupling the two detectors (e.g.,wherein the first detector 802 is a face detector and the seconddetector 812 is a pedestrian detector).

In at least one example, the output 814 can be provided to other vehiclesystem(s) 818 of the computing system onboard the vehicle 806. Suchsystem(s) can include a prediction system 820, a planning system 822,and so on. Additional systems are described below with reference to FIG.3. In at least one example, the output 814 can be provided to the othervehicle system(s) 818 for making determinations with respect tocontrolling the vehicle 806, for instance, for determining a trajectoryalong which the vehicle 806 is to drive and causing the vehicle 806 todrive along the trajectory. In some examples, the output 814 can be usedby the prediction system 820 for predicting how an object in anenvironment may move in the future. In additional or alternativeexamples, the output 814 can be used by the planning system 822 todetermine a trajectory along which the vehicle 806 is to travel.Additional details associated with the prediction system 820 and theplanning system 822 are described below with reference to FIG. 3.

FIG. 1 illustrates an example process 100 for performing techniquesdescribed herein. FIG. 1 is similar to FIG. 8 but describes a particularapplication of techniques described above in FIG. 8. In FIG. 1, a headdetector 102, which can be associated with a perception system asdescribed below, can receive sensor data, such as image data 104. Theimage data 104 can include one or more frames of images of anenvironment within which a vehicle 106 is positioned. In at least oneexample, the head detector 102 (and the perception system) can beonboard the vehicle 106. In at least one example, the vehicle 106 can bean autonomous vehicle configured to operate according to a Level 5classification issued by the U.S. National Highway Traffic SafetyAdministration, which describes a vehicle capable of performing allsafety-critical functions for the entire trip, with the driver (oroccupant) not being expected to control the vehicle at any time. In suchan example, since the vehicle 106 can be configured to control allfunctions from start to stop, including all parking functions, it can beunoccupied. This is merely an example, and the systems and methodsdescribed herein can be incorporated into any ground-borne, airborne, orwaterborne vehicle, including those ranging from vehicles that need tobe manually controlled by a driver at all times, to those that arepartially or fully autonomously controlled. That is, in the illustratedexample, the vehicle 106 is an autonomous vehicle; however, the vehicle106 could be any other type of vehicle.

In at least one example, the head detector 102 can receive the imagedata 104 and can analyze the image data 104 using a machine learnedmodel. As described above, in at least one example, the head detector102 can be part of a computing system onboard the vehicle 106 that canutilize a neural network for detecting head(s) in the image data 104.However, in additional or alternative examples, the head detector 102can use any other type of machine learned model, as described below.Examples of machine learned models that can be used by the head detector102 include, but are not limited to, FaceMatch, Face Everything,Fisherfaces, DeepID Test, Tensorflow, OpenFace, TinyFaces, etc. The headdetector 102 can generate an output 108 based at least in part onanalyzing the image data 104 with the machine learned model. In at leastone example, the output 108 can include the image data 104 with one ormore indications 110 of whether a head was detected by the head detector102. In some examples, the indication(s) 110 can be bounding box(es)that surround region(s) of interest corresponding to head(s) in theimage data 104 and/or corresponding certainty information. In otherexamples, any other type of indication can be used to indicate whether ahead was detected by the head detector 102. In at least one example, theindication(s) 110 can be associated with information including anx-position (global and/or local position), a y-position (global and/orlocal position), a z-position (global and/or local position), anorientation (e.g., a roll, pitch, yaw), an extent of the object (e.g.,height (h), width (w), etc.), an indication of whether a head wasdetected (e.g., a binary indication (i.e., head detected (H) or not(N)), a probability, etc.), etc. In some examples, the information canbe used to generate the bounding box(es) and/or other indications.Additional details are illustrated in FIG. 2A.

In some examples, the head detector 102 can be associated with amulti-class classifier, which can analyze the image data 104 and outputa plurality of classifications with corresponding certainty information(e.g., probability of a detected object corresponding to a particularclassification). In such an example, a machine learned model can betrained to classify heads into classifications such as a head of apedestrian, a head of a cyclist, a head on a billboard, etc. Suchclassifications and/or corresponding certainty information can be usefulfor determining indications of heads, as described herein.

In at least one example, the output 108 of the head detector 102 can beinput into a pedestrian detector 112, which can determine whether apedestrian is present in the environment within which the vehicle 106 ispresented and/or information associated with the pedestrian. That is,the pedestrian detector 112 can analyze the output 108 to determinewhether a pedestrian is present in the environment within which thevehicle 106 is presented and/or information associated with thepedestrian. In at least one example, the pedestrian detector 112 can beassociated with a perception system of the vehicle 106, as describedbelow. In some examples, the pedestrian detector 112 can use one or morestatistical models to determine whether a pedestrian is present in anenvironment (e.g., based on a size and/or position of the bounding boxof the output 108, estimating (based on a statistical likelihood) aproposed region for a pedestrian in the image). In additional oralternative examples, the pedestrian detector 112 can use one or moremachine learned models, which can analyze the output 108 and, in someexamples, additional data as described below, to determine whether apedestrian is present in an environment. In some examples, the imagedata 104 can be input directly into the pedestrian detector 112 inaddition to the head detector 102. Additional details are describedbelow with reference to FIGS. 4-7.

The pedestrian detector 112 can generate an output 114 that is based atleast partly on the output 108. In at least one example, the output 114of the pedestrian detector 112 can include one or more indications 116that are associated with regions of interest corresponding to identifiedpedestrian(s) in the environment within which the vehicle 106 ispositioned. In some examples, the indication(s) 116 can be boundingbox(es) that surround region(s) of interest corresponding topedestrian(s) in the image data 104. In other examples, any other typeof indication can be used to indicate that a pedestrian was detected inthe environment by the pedestrian detector 112. In at least one example,the indication(s) 116 can be associated with information including anx-position (global and/or local position), a y-position (global and/orlocal position), a z-position (global and/or local position), anorientation (e.g., a roll, pitch, yaw), an object type (e.g., aclassification), a velocity of the object, an acceleration of theobject, an extent of the object (e.g., height (h), width (w), etc.),etc. In some examples, the information can be used to generate thebounding box(es) and/or other indications. Additional details areillustrated in FIG. 2B.

In at least one example, the output 114 can be provided to other vehiclesystem(s) 118 of the computing system onboard the vehicle 106. Suchsystem(s) can include a prediction system 120, a planning system 122,and so on. Additional systems are described below with reference to FIG.3. In at least one example, the output 114 can be provided to the othervehicle system(s) 118 for making determinations with respect tocontrolling the vehicle 106, for instance, for determining a trajectoryalong which the vehicle 106 is to drive and causing the vehicle 106 todrive along the trajectory. In some examples, the output 114 can be usedby the prediction system 120 for predicting how an object in anenvironment may move in the future. In additional or alternativeexamples, the output 114 can be used by the planning system 122 todetermine a trajectory along which the vehicle 106 is to travel.Additional details associated with the prediction system 120 and theplanning system 122 are described below with reference to FIG. 3.

FIGS. 2A and 2B illustrate additional details associated with theoutputs 108 and 114, respectively.

FIG. 2A illustrates additional details associated with an output of ahead detection process. That is, FIG. 2A illustrates additional detailsassociated with the output 108 in FIG. 1. As described above, the headdetector 102 can generate an output 108 based at least in part onanalyzing the image data 104 with the machine learned model. In at leastone example, the output 108 can include the image data 104 with one ormore indications 110 of whether a head was detected by the head detector102. The portion 200 of the output 108 illustrated in FIG. 2Aillustrates a first indication 202(A) associated with a first head and asecond indication 202(B) associated with a second head (collectively,indication(s) 202). In some examples, the indication(s) 202 can bebounding box(es) that surround region(s) of interest corresponding tohead(s) in the image data 104 and/or corresponding certaintyinformation. In other examples, any other type of indication can be usedto indicate whether a head was detected by the head detector 102. In atleast one example, the indication(s) 202 can be associated withinformation including an x-position (global and/or local position), ay-position (global and/or local position), a z-position (global and/orlocal position), an orientation (e.g., a roll, pitch, yaw), an extent ofthe object (e.g., height (h), width (w), etc.), an indication of whethera head was detected (e.g., a binary indication (i.e., head detected (H)or not (N)), a probability, etc.), etc. In FIG. 2A, the information204(A) and 204(B) is illustrated as being displayed proximate to therespective indication(s) 202.

FIG. 2B illustrates additional details associated with an output of apedestrian detection process. That is, FIG. 2B illustrates additionaldetails associated with the output 114 in FIG. 1. As described above,the pedestrian detector 112 can generate an output 114. In at least oneexample, the output 114 of the pedestrian detector 112 can include oneor more indicators 116 that are associated with regions of interestcorresponding to identified pedestrian(s) in the environment withinwhich the vehicle 106 is positioned. The portion 206 of the output 114illustrated in FIG. 2B illustrates a third indication 208(A) associatedwith a first pedestrian and a fourth indication 208(B) associated with asecond pedestrian (collectively, indication(s) 208). In some examples,the indication(s) 208 can be bounding box(es) that surround region(s) ofinterest corresponding to pedestrian(s) in the image data 104. In otherexamples, any other type of indication can be used to indicate that apedestrian was determined to be present in the environment by thepedestrian detector 112. In at least one example, the indication(s) 208can be associated with information including an x-position (globaland/or local position), a y-position (global and/or local position), az-position (global and/or local position), an orientation (e.g., a roll,pitch, yaw), an object type (e.g., a classification), a velocity of theobject, an acceleration of the object, an extent of the object (e.g.,height (h), width (w), etc.), etc., as well as any certaintiesassociated therewith. In FIG. 2B, the information 210(A) and 210(B) isillustrated as being displayed proximate to the indication(s) 208.

FIG. 2B additionally includes the indication(s) 202 output via the headdetector 102 and the information 204(A) and 204(B) associated therewith.That is, FIG. 2B illustrates that the output 114 includes both theindication(s) 202 and the indication(s) 208, as well as the information204(A), 204(B), 210(A), and 210(B) associated with each. In analternative example, the indication(s) 202 and/or associated information204(A) and 204(B) can be omitted.

As described above, techniques described herein enable pedestrians to bedetected with more accuracy than what is available with currenttechniques. Conventional object detection techniques may not haveidentified the pedestrian in the image data 104 due to the pedestrian'spositioning and/or the additional objects surrounding the pedestrian(e.g., the stroller and the child on the bike). Further, the child onthe bike may be identified as a cyclist. However, by using headdetection to determine that pedestrians are present in image data,techniques described herein can determine pedestrians are presentdespite variations and/or oddities described above.

FIG. 3 is a block diagram illustrating an example system 300 forperforming techniques described herein. In at least one example, avehicle 302 can include one or more vehicle computing devices 304, oneor more sensor systems 306, one or more emitters 308, one or morecommunication connections 310, at least one direct connection 312, andone or more drive systems 314. As described above, the vehicle 302 canbe an autonomous vehicle configured to operate according to a Level 5classification issued by the U.S. National Highway Traffic SafetyAdministration, which describes a vehicle capable of performing allsafety-critical functions for the entire trip, with the driver (oroccupant) not being expected to control the vehicle at any time. In suchan example, since the vehicle 302 can be configured to control allfunctions from start to stop, including all parking functions, it can beunoccupied. This is merely an example, and the systems and methodsdescribed herein can be incorporated into any ground-borne, airborne, orwaterborne vehicle, including those ranging from vehicles that need tobe manually controlled by a driver at all times, to those that arepartially or fully autonomously controlled. That is, in the illustratedexample, the vehicle 302 is an autonomous vehicle; however, the vehicle302 could be any other type of vehicle.

The vehicle computing device(s) 304 can include processor(s) 316 andmemory 318 communicatively coupled with the processor(s) 316. In theillustrated example, the memory 318 of the vehicle computing device(s)304 stores a localization system 320, a perception system 322, aprediction system 324, a planning system 326, and one or more systemcontrollers 328. Additionally, the memory 318 can include a storage 330,which can store map(s), model(s), etc. A map can be any number of datastructures modeled in two dimensions, three dimensions, or N dimensionsthat are capable of providing information about an environment, such as,but not limited to, topologies (such as intersections), streets,mountain ranges, roads, terrain, and the environment in general. Mapscan be associated with real environments or simulated environments.Model(s) can include machine-trained models, as described below.

In at least one example, the localization system 320 can determine apose (position and orientation) of the vehicle 302 in relation to alocal and/or global map based at least in part on sensor data receivedfrom the sensor system(s) 306 and/or map data associated with a map(e.g., of the map(s)). In at least one example, the localization system320 can include, or be associated with a calibration system that iscapable of performing operations for calibrating (determining variousintrinsic and extrinsic parameters associated with any one or more ofthe sensor system(s) 306), localizing, and mapping substantiallysimultaneously. Additional details associated with such a system aredescribed in U.S. patent application Ser. No. 15/675,487, filed on Aug.11, 2017, which is related to U.S. patent application Ser. No.15/674,853, filed on Aug. 11, 2017, the entire contents of both of whichare incorporated by reference herein.

In at least one example, the perception system 322 can perform objectdetection, segmentation, and/or classification based at least in part onsensor data received from the sensor system(s) 306. In at least oneexample, the perception system 322 can receive raw sensor data (e.g.,from the sensor system(s) 306). In at least one example, the perceptionsystem 322 can receive image data and can utilize one or more imageprocessing algorithms to perform object detection, segmentation, and/orclassification with respect to object(s) identified in the image data.In some examples, the perception system 322 can associate a bounding box(or otherwise an instance segmentation) with an identified object andcan associate a confidence score associated with a classification of theidentified object with the identified object. In some examples, objects,when rendered via a display, can be colored based on their perceivedclass. In at least one example, the perception system 322 can performoperations described herein as being performed by the detectors 323,which can correspond to the detectors described above in FIGS. 1 and 8.In other examples, similar processes (detection, classification,segmentation, etc.) may be performed by the perception system 322 and/orby another system that provides input to the perception system 322. Theperception system 322 can perform similar processes for one or moreother modalities (e.g., light detection and ranging (LIDAR), radiodetection and ranging (RADAR), Time of Flight (ToF) systems, etc.).

The prediction system 324, which can correspond to the prediction system120 and 820, described above with reference to FIGS. 1 and 8, can accesssensor data from the sensor system(s) 306, map data associated with amap (e.g., of the map(s) which can be in storage 330), and/or perceptiondata output from the perception system 322 (e.g., processed sensordata), and can output predictions associated with one or more objectswithin the environment of the vehicle 302. In at least one example, theplanning system 326, which can correspond to the planning system 122 and822, described above with reference to FIGS. 1 and 8, can determineroutes and/or trajectories to use to control the vehicle 302 based atleast in part on sensor data received from the sensor system(s) 306and/or any determinations made by the perception system 322 and/orprediction system 324. Additional details of localization systems,perception systems, prediction systems, and/or planning systems that areusable can be found in U.S. Pat. No. 9,612,123, issued on Apr. 4, 2017,and U.S. patent application Ser. No. 15/632,208, filed Jun. 23, 2017,now known as U.S. Pat. No. 10,353,390, issued on Jul. 16, 2019, theentire contents of both of which are incorporated by reference herein.In some examples (e.g., where the vehicle 302 is not an autonomousvehicle), one or more of the aforementioned systems can be omitted fromthe vehicle 302. While the systems described above are illustrated as“onboard” the vehicle 302, in other implementations, the systems can beremotely located and/or accessible to the vehicle 302. Furthermore,while the systems are described above as “systems,” such systems cancomprise one or more components for performing operations attributed toeach of the systems.

In at least one example, the localization system 320, the perceptionsystem 322, the prediction system 324, and/or the planning system 326can process sensor data, as described above, and can send theirrespective outputs over network(s) 332, to computing device(s) 334. Inat least one example, the localization system 320, the perception system322, the prediction system 324, and/or the planning system 326 can sendtheir respective outputs to the computing device(s) 334 at a particularfrequency, after a lapse of a predetermined period of time, in nearreal-time, etc.

In at least one example, the vehicle computing device(s) 304 can includeone or more system controllers 328, which can be configured to controlsteering, propulsion, braking, safety, emitters, communication, andother systems of the vehicle 302. These system controller(s) 328 cancommunicate with and/or control corresponding systems of the drivesystem(s) 314 and/or other systems of the vehicle 302.

In at least one example, the sensor system(s) 306 can include LIDARsensors, RADAR sensors, ultrasonic transducers, sound navigation andranging (SONAR) sensors, location sensors (e.g., global positioningsystem (GPS), compass, etc.), inertial sensors (e.g., inertialmeasurement units, accelerometers, magnetometers, gyroscopes, etc.),cameras (e.g., RGB, R, intensity, depth, etc.), wheel encoders,microphones, environment sensors (e.g., temperature sensors, humiditysensors, light sensors, pressure sensors, etc.), ToF sensors, etc. Thesensor system(s) 306 can include multiple instances of each of these orother types of sensors. For instance, the LIDAR sensors can includeindividual LIDAR sensors located at the corners, front, back, sides,and/or top of the vehicle 302. As another example, the camera sensorscan include multiple cameras positioned at various locations about theexterior and/or interior of the vehicle 302. The sensor system(s) 306can provide input to the vehicle computing device(s) 304. In someexamples, the sensor system(s) 306 can preprocess at least some of thesensor data prior to sending the sensor data to the vehicle computingdevice(s) 304. In at least one example, the sensor system(s) 306 cansend sensor data, via the network(s) 332, to the computing device(s) 334at a particular frequency, after a lapse of a predetermined period oftime, in near real-time, etc.

The vehicle 302 can also include one or more emitters 308 for emittinglight and/or sound, as described above. The emitter(s) 308 in thisexample include interior audio and visual emitters to communicate withpassengers of the vehicle 302. By way of example and not limitation,interior emitters can include speakers, lights, signs, display screens,touch screens, haptic emitters (e.g., vibration and/or force feedback),mechanical actuators (e.g., seatbelt tensioners, seat positioners,headrest positioners, etc.), and the like. The emitter(s) 308 in thisexample also include exterior emitters. By way of example and notlimitation, the exterior emitters in this example include light emitters(e.g., indicator lights, signs, light arrays, etc.) to visuallycommunicate with pedestrians, other drivers, other nearby vehicles,etc., one or more audio emitters (e.g., speakers, speaker arrays, horns,etc.) to audibly communicate with pedestrians, other drivers, othernearby vehicles, etc., etc. In at least one example, the emitter(s) 308can be positioned at various locations about the exterior and/orinterior of the vehicle 302.

The vehicle 302 can also include communication connection(s) 310 thatenable communication between the vehicle 302 and other local or remotecomputing device(s). For instance, the communication connection(s) 310can facilitate communication with other local computing device(s) on thevehicle 302 and/or the drive system(s) 314. Also, the communicationconnection(s) 310 can allow the vehicle to communicate with other nearbycomputing device(s) (e.g., other nearby vehicles, traffic signals,etc.). The communications connection(s) 310 also enable the vehicle 302to communicate with a remote teleoperations computing device or otherremote services.

The communications connection(s) 310 can include physical and/or logicalinterfaces for connecting the vehicle computing device(s) 304 to anothercomputing device or a network, such as network(s) 332. For example, thecommunications connection(s) 310 can enable Wi-Fi-based communicationsuch as via frequencies defined by the IEEE 802.11 standards, shortrange wireless frequencies such as BLUETOOTH®, or any suitable wired orwireless communications protocol that enables the respective computingdevice to interface with the other computing device(s).

The direct connection 312 can directly connect the drive system(s) 314and other systems of the vehicle 302.

In at least one example, the vehicle 302 can include drive system(s)314. In some examples, the vehicle 302 can have a single drive system314. In at least one example, if the vehicle 302 has multiple drivesystems 314, individual drive systems 314 can be positioned on oppositeends of the vehicle 302 (e.g., the front and the rear, etc.). In atleast one example, the drive system(s) 314 can include sensor system(s)to detect conditions of the drive system(s) 314 and/or the surroundingsof the vehicle 302. By way of example and not limitation, the sensorsystem(s) can include wheel encoder(s) (e.g., rotary encoders) to senserotation of the wheels of the drive module, inertial sensors (e.g.,inertial measurement units, accelerometers, gyroscopes, magnetometers,etc.) to measure position and acceleration of the drive module, camerasor other image sensors, ultrasonic sensors to acoustically detectobjects in the surroundings of the drive module, LIDAR sensors, RADARsensors, etc. Some sensors, such as the wheel encoder(s), can be uniqueto the drive system(s) 314. In some cases, the sensor system(s) on thedrive system(s) 314 can overlap or supplement corresponding systems ofthe vehicle 302 (e.g., sensor system(s) 306).

The drive system(s) 314 can include many of the vehicle systems,including a high voltage battery, a motor to propel the vehicle 302, aninverter to convert direct current from the battery into alternatingcurrent for use by other vehicle systems, a steering system including asteering motor and steering rack (which can be electric), a brakingsystem including hydraulic or electric actuators, a suspension systemincluding hydraulic and/or pneumatic components, a stability controlsystem for distributing brake forces to mitigate loss of traction andmaintain control, an HVAC system, lighting (e.g., lighting such ashead/tail lights to illuminate an exterior surrounding of the vehicle),and one or more other systems (e.g., cooling system, safety systems,onboard charging system, other electrical components such as a DC/DCconverter, a high voltage junction, a high voltage cable, chargingsystem, charge port, etc.). Additionally, the drive system(s) 314 caninclude a drive module controller which can receive and preprocess datafrom the sensor system(s) and to control operation of the variousvehicle systems. In some examples, the drive module controller caninclude processor(s) and memory communicatively coupled with theprocessor(s). The memory can store one or more modules to performvarious functionalities of the drive system(s) 314. Furthermore, thedrive system(s) 314 also include communication connection(s) that enablecommunication by the respective drive module with other local or remotecomputing device(s).

In FIG. 3, the vehicle computing device(s) 304, sensor system(s) 306,emitter(s) 308, and the communication connection(s) 310 are shownonboard the vehicle 302. However, in some examples, the vehiclecomputing device(s) 304, sensor system(s) 306, emitter(s) 308, and thecommunication connection(s) 310 can be implemented outside of an actualvehicle (i.e., not onboard the vehicle 302).

As described above, the vehicle 302 can send sensor data to thecomputing device(s) 334, via the network(s) 332. In some examples, thevehicle 302 can send raw sensor data to the computing device(s) 334. Inother examples, the vehicle 302 can send processed sensor data and/orrepresentations of sensor data to the computing device(s) 334 (e.g.,data output from the localization system 320, the perception system 322,the prediction system 324, and/or the planning system 326). In someexamples, the vehicle 302 can send sensor data to the computingdevice(s) 334 at a particular frequency, after a lapse of apredetermined period of time, in near real-time, etc.

The computing device(s) 334 can receive the sensor data (raw orprocessed) from the vehicle 302 and/or other data collection devices(which can include other vehicles like 302), as well as data from one ormore third party sources and/or systems. In at least one example, thecomputing device(s) 334 can include processor(s) 336 and memory 338communicatively coupled with the processor(s) 336. In the illustratedexample, the memory 338 of the computing device(s) 334 stores a trainingsystem 340, a map(s) storage 342 (e.g., storing one or more maps), atraining data storage 344 (e.g., storing training data accessible to thetraining system 340), and a model(s) storage 346 (e.g., models output bythe training system 340). In some examples, one or more of the systemsand/or storage repositories can be associated with the vehicle 302instead of, or in addition to, being associated with the memory 338 ofthe computing device(s) 334.

In at least one example, the training system 340 can train datamodel(s), which can be used for determining the presence of objects, orportions thereof, in image data as described herein. In at least oneexample, the training system 340 can train data model(s) using machinelearning techniques. For instance, in at least one example, aconvolutional neural network (CNN) can be trained using training data,which can include image data. Such training data can include image dataincluding objects, pedestrians, heads, and so on (e.g., all of which canbe considered regions) and, in some examples, image data where objects,pedestrians, heads, and so on are not present. In some examples, thetraining data can be labeled. For instance, in some examples, lowvariance regions and high variance regions can be labeled, andcorrespondence between low variance regions and high variance regionsindicated. In some examples, high variance regions can be labeled, butlow variance regions may not be labeled. In some examples, low varianceregions can be labeled, but high variance regions may not be labeled. Insome examples, the training data may not be labeled.

In at least one example, the training system 340 can train, usingmachine learning, a model for detecting objects (e.g., pedestrians,cyclists, other vehicles, etc.) in image data (e.g., an objectdetector). In such an example, the training data can include imagestagged, or otherwise annotated, to indicate where an object is located,with an indication of a classification of the object. The CNN cananalyze the training data to learn what constitutes an object in imagedata and can use such knowledge to detect like-objects in other imagedata (e.g., newly received image data). That is, the CNN can be trainedto detect objects in image data. In at least one example, such a machinelearned model can be used by the perception system 322 to detect objectsin image data. In some examples, the machine learned model can output anindication (e.g., a bounding box) associated with a region of interestcorresponding to a detected object and/or information associated withthe detected object, as described above.

Furthermore, in at least one example, the CNN can be trained to classifyobjects in image data. For example, the training system 340 can train,using machine learning, a model for classifying objects in image data.In such an example, the training data can include images tagged, orotherwise annotated, to indicate where an object is located, with anindication of a classification of the object. In at least one example,the machine learned model can be trained to output a plurality ofclassifications associated with a detected object. For example, themachine learned model can be a multi-class classifier that is trained tooutput a probability that an object is associated with individualclasses of the multi-class classifier. In some examples, theclassifications can be ranked from highest probability to lowestprobability.

In at least one example, the training system 340 can train, usingmachine learning, a model for detecting high variance regions in imagedata (e.g., the second detector 812 described above with reference toFIG. 8). In some examples, the training data can include images tagged,or otherwise annotated, to indicate where a high variance region islocated. In other examples, high variance regions may not be tagged orotherwise annotated. As described above, a high variance region is aregion where multiple instances of the region (e.g., regions classifiedwith a same bounding box) in input data (e.g., image data, a batch ofimage data, an input space) are compared, a variance metric associatedwith the multiple instances of the region can meet or exceed athreshold. Such a variance metric can represent information inside of abounding box associated with a region. In some examples, the variancemetric can be based on entropy, pixel intensity, aspect ratios betweenboxes, and the like. In some examples, the training data can be outputfrom another machine learned model. In at least one example, thetraining data can be an intermediate output of features. The CNN cananalyze the training data to learn what constitutes a high varianceregion in image data and can use such knowledge to detect high varianceregions in other image data (e.g., newly received image data). That is,the CNN can be trained to detect high variance regions in image data.Additional details associated with training a machine learned model fordetecting a high variance region are described below with reference toFIG. 13. In at least one example, such a machine learned model can beused by the perception system 322 and/or the second detector 812 todetect high variance regions in image data. In some examples, themachine learned model can output an indication (e.g., a bounding box)associated with a region of interest corresponding to a detected highvariance region and/or information associated with the detected highvariance region, as described above.

Further, the training system 340 can train, using machine learning, amodel for detecting low variance regions in image data (e.g., the firstdetector 802 described above with reference to FIG. 8). In such anexample, the training data can include images tagged, or otherwiseannotated, to indicate where a low variance region is located. In someexamples, a higher variance region or input data can be projected into alower dimensional space, whereby the result can be a space having alower variance. Such a result can be tagged as, or otherwise indicatedto be, a low variance region. As described above, a low variance regionis a region where multiple instances of the region (e.g., regionsclassified with a same bounding box) in input data (e.g., image data, abatch of image data, an input space) are compared, a variance metricassociated with the multiple instances of the region is below athreshold. Such a variance metric can represent information inside of abounding box associated with a region. In some examples, the variancemetric can be based on entropy, pixel intensity, aspect ratios betweenboxes, and the like. The CNN can analyze the training data to learn whatconstitutes a low variance region in image data and can use suchknowledge to detect low variance regions in other image data (e.g.,newly received image data). That is, the CNN can be trained to detectlow variance regions in image data. Additional details associated withtraining a machine learned model for detecting a low variance region aredescribed below with reference to FIG. 13. In at least one example, sucha machine learned model can be used by the perception system 322 and/orthe first detector 802 to detect low variance regions in image data. Insome examples, the machine learned model can output an indication (e.g.,a bounding box) associated with a region of interest corresponding to adetected low variance region and/or information associated with thedetected high variance region, as described above.

In another example, the training system 340 can train, using machinelearning, a model for detecting low variance regions in image data(e.g., the first detector 802 described above with reference to FIG. 8)using images tagged, or otherwise annotated, to indicate where a highvariance region is located (but not a low variance region). The CNN cananalyze the training data to learn what constitutes a low varianceregion in association with tagged high variance regions in image dataand can use such knowledge to detect low variance regions in other imagedata (e.g., newly received image data). That is, the CNN can be trainedto detect low variance regions in image data. Additional detailsassociated with training a machine learned model for detecting a highvariance region are described below with reference to FIG. 13. In atleast one example, such a machine learned model can be used by theperception system 322 and/or the first detector 802 to detect lowvariance regions in image data. In some examples, the machine learnedmodel can output an indication (e.g., a bounding box) associated with aregion of interest corresponding to a detected low variance regionand/or information associated with the detected high variance region, asdescribed above.

In some examples, the training system 340 can train, using machinelearning, a model for detecting high variance regions, based on lowvariance features, by modelling an end-to-end optimization problem. Insuch examples, the low variance outputs may not be explicitly modeled,but instead, a model can be trained to detect high variance regionsbased at least in part on implicit low variance regions. In an example,input data (e.g., image data) can be projected (e.g., via a neuralnetwork) into a feature space that represents the input data. Suchfeature space includes high-level information for object detection. Inan example, another neural network can be used to reconstruct the inputdata (e.g., image data) based on features in the feature space. That is,the features can be projected into an image space (e.g., using a neuralnetwork). In at least one example, a loss function can be employed toenforce consistency between the input data and the reconstructed inputdata. That is, a loss function can be employed by the training system340 to train a model to focus on low variance features for detectinghigh variance regions (e.g., explicitly modelling high varianceregions). Additional details associated with training a machine learnedmodel for detecting a high variance region are described below withreference to FIG. 13.

In any such examples, the machine learned model(s) (which can beassociated with the detector(s) described above with reference to FIGS.1 and 8) may be trained independently and, in some examples,pre-informed of relationships between low variance regions and highvariance regions. In some examples, the machine learned model(s) can betrained end-to-end. When trained end-to-end, such training may enforcethe coupling between low variance regions which inform the ultimateoutput. In some examples, training such a machine learned model todetect a low variance region may comprise evaluating andback-propagating based at least in part on a loss function whichdetermines a measure of the variance of an associated region in theinput space. Additional details associated with training a machinelearned model for detecting a high variance region are described belowwith reference to FIG. 13.

In at least those examples in which both the first detector 802 and thesecond detector 812 both comprise machine learned models, each maycomprise a separate network, may comprise different aspects of a commonnetwork, or may otherwise comprise a combination of multiple networks.In any of the examples provided herein, any of the machine learnedmodels may be trained end-to-end, jointly, disjointly, or the like. Insuch examples where the models are trained end-to-end or jointly, commonfeatures may be leveraged by one network to improve outputs of the other(e.g., where features learned by the first detector 802 to detect lowvariance regions are used to predict high variance regions in the seconddetector 812).

In another example, the training system 340 can train, using machinelearning, a model for detecting pedestrians in image data (e.g., apedestrian detector). In some examples, the training data can includeimages tagged, or otherwise annotated, to indicate where a pedestrian islocated. In additional or alternative examples, the training data can beoutput from another machine learned model. In at least one example, thetraining data can be an intermediate output of features. The CNN cananalyze the training data to learn what constitutes a pedestrian inimage data and can use such knowledge to detect pedestrians in otherimage data (e.g., newly received image data). That is, the CNN can betrained to detect pedestrians in image data. In at least one example,such a machine learned model can be used by the perception system 322and/or the pedestrian detector 112 to detect pedestrians in image data.In some examples, the machine learned model can output an indication(e.g., a bounding box) associated with a region of interestcorresponding to a detected pedestrian and/or information associatedwith the detected pedestrian, as described above.

Further, the training system 340 can train, using machine learning, amodel for detecting heads in image data (e.g., a head detector). In suchan example, the training data can include images tagged, or otherwiseannotated, to indicate where a head is located. The CNN can analyze thetraining data to learn what constitutes a head in image data and can usesuch knowledge to detect heads in other image data (e.g., newly receivedimage data). That is, the CNN can be trained to detect heads in imagedata. In at least one example, such a machine learned model can be usedby the perception system 322 and/or the head detector 102 to detectheads in image data. In some examples, the machine learned model canoutput an indication (e.g., a bounding box) associated with a region ofinterest corresponding to a detected head and/or information associatedwith the detected pedestrian, as described above.

In at least those examples in which both the head detector 102 and thepedestrian detector 112 both comprise machine learned models, each maycomprise a separate network, may comprise different aspects of a commonnetwork, or may otherwise comprise a combination of multiple networks.In any of the examples provided herein, any of the machine learnedmodels may be trained end-to-end, jointly, disjointly, or the like. Insuch examples where the models are trained end-to-end or jointly, commonfeatures may be leveraged by one network to improve outputs of the other(e.g., where features learned by the head detector 102 to detect headsare used to predict pedestrians in pedestrian detector 112).

In some examples, outputs from other machine learned models can be usedas part of the training data for training and/or updating the datamodel(s) described herein. For example, outputs of the head detector 102and/or an object detector can be used to train the machine learned modelto identify pedestrians (e.g., the pedestrian detector). In anadditional or alternative example, outputs of the head detector 102 canbe used to train the machine learned model to identify objects (e.g.,the object detector), so that the object detector includes theclassification of “head” in the plurality of classifications output inassociation with a detected object.

Although discussed in the context of CNNs above, any type of machinelearning can be used consistent with this disclosure. For example,machine learning algorithms for training machine learned model(s) caninclude, but are not limited to, regression algorithms (e.g., ordinaryleast squares regression (OLSR), linear regression, logistic regression,stepwise regression, multivariate adaptive regression splines (MARS),locally estimated scatterplot smoothing (LOESS)), example-basedalgorithms (e.g., ridge regression, least absolute shrinkage andselection operator (LASSO), elastic net, least-angle regression (LARS)),decisions tree algorithms (e.g., classification and regression tree(CART), iterative dichotomiser 3 (ID3), Chi-squared automaticinteraction detection (CHAID), decision stump, conditional decisiontrees), Bayesian algorithms (e.g., naïve Bayes, Gaussian naïve Bayes,multinomial naïve Bayes, average one-dependence estimators (AODE),Bayesian belief network (BNN), Bayesian networks), clustering algorithms(e.g., k-means, k-medians, expectation maximization (EM), hierarchicalclustering), association rule learning algorithms (e.g., perceptron,back-propagation, hopfield network, Radial Basis Function Network(RBFN)), deep learning algorithms (e.g., Deep Boltzmann Machine (DBM),other Deep Belief Networks (DBN), Artificial Neural Network (ANN),Stacked Auto-Encoders), Dimensionality Reduction Algorithms (e.g.,Principal Component Analysis (PCA), Principal Component Regression(PCR), Partial Least Squares Regression (PLSR), Sammon Mapping,Multidimensional Scaling (MDS), Projection Pursuit, Linear DiscriminantAnalysis (LDA), Mixture Discriminant Analysis (MDA), QuadraticDiscriminant Analysis (QDA), Flexible Discriminant Analysis (FDA)), SVM(support vector machine), supervised learning, unsupervised learning,semi-supervised learning, etc.

The resulting data model(s) can be stored in the model(s) storage 346and/or the storage 330 on the vehicle 302 and can be accessed by theperception system 322 for detecting and/or classifying objects.

The processor(s) 316 of the vehicle 302 and the processor(s) 336 of thecomputing device(s) 334 can be any suitable processor capable ofexecuting instructions to process data and perform operations asdescribed herein. By way of example and not limitation, the processor(s)316 and 336 can comprise one or more Central Processing Units (CPUs),Graphics Processing Units (GPUs), or any other device or portion of adevice that processes electronic data to transform that electronic datainto other electronic data that can be stored in registers and/ormemory. In some examples, integrated circuits (e.g., ASICs, etc.), gatearrays (e.g., FPGAs, etc.), and other hardware devices can also beconsidered processors in so far as they are configured to implementencoded instructions.

Memory 318 and 338 are examples of non-transitory computer-readablemedia. Memory 318 and 338 can store an operating system and one or moresoftware applications, instructions, programs, and/or data to implementthe methods described herein and the functions attributed to the varioussystems. In various implementations, the memory can be implemented usingany suitable memory technology, such as static random access memory(SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory,or any other type of memory capable of storing information. Thearchitectures, systems, and individual elements described herein caninclude many other logical, programmatic, and physical components, ofwhich those shown in the accompanying figures are merely examples thatare related to the discussion herein.

It should be noted that while FIG. 3 is illustrated as a distributedsystem, in alternative examples, systems of the vehicle 302 can beassociated with the computing device(s) 334 and/or systems of thecomputing device(s) 334 can be associated with the vehicle 302. That is,the vehicle 302 can perform one or more of the functions associated withthe computing device(s) 334, and vice versa.

FIGS. 4-7 and 9-12 are flowcharts showing example methods involvingtechniques as described herein. The processes illustrated in FIGS. 4-7and 9-12 are described with reference to the system 300 shown in FIG. 3for convenience and ease of understanding. However, the processesillustrated in FIGS. 4-7 and 9-12 are not limited to being performedusing the system 300. Moreover, the system 300 described herein is notlimited to performing the processes illustrated in FIGS. 4-7 and 9-12.

The processes 400-700 and 900-1200 are illustrated as collections ofblocks in logical flow graphs, which represent sequences of operationsthat can be implemented in hardware, software, or a combination thereof.In the context of software, the blocks represent computer-executableinstructions stored on one or more computer-readable storage media that,when executed by processor(s), perform the recited operations.Generally, computer-executable instructions include routines, programs,objects, components, data structures, and the like that performparticular functions or implement particular abstract data types. Theorder in which the operations are described is not intended to beconstrued as a limitation, and any number of the described blocks can becombined in any order and/or in parallel to implement the processes. Insome embodiments, one or more blocks of the process can be omittedentirely. Moreover, the methods 400-700 and 900-1200 can be combined inwhole or in part with each other or with other methods.

FIG. 9 illustrates an example process 900 for using sub-region detectiontechniques for improving detections (including detections associatedwith high variance input data), as described herein.

At operation 902, the perception system 322 receives image data. Asdescribed above, a vehicle 302 can include one or more sensor systems306. The sensor system(s) 306 can include LIDAR sensors, RADAR sensors,ultrasonic transducers, SONAR sensors, location sensors (e.g., GPS,compass, etc.), inertial sensors (e.g., inertial measurement units,accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB,IR, intensity, depth, etc.), wheel encoders, microphones, environmentsensors (e.g., temperature sensors, humidity sensors, light sensors,pressure sensors, etc.), ToF sensors, etc. The sensor system(s) 306 canprovide input to the vehicle computing device(s) 304. In at least oneexample, the sensor system(s) 306 can provide image data to theperception system 322.

At operation 904, the perception system 322 analyzes the image datausing a first machine learned model trained to detect low varianceregions. As described above with reference to FIG. 3, the perceptionsystem 322 can perform object detection, segmentation, and/orclassification based at least in part on sensor data received from thesensor system(s) 306. In at least one example, the perception system 322can receive the image data and can utilize one or more image processingalgorithms to perform object detection, segmentation, and/orclassification with respect to object(s) identified in the image data.In at least one example, the perception system 322 can perform lowvariance region detection using a first detector 802. As describedabove, in at least one example, the first detector 802 can utilize aneural network for detecting low variance region(s) associated with theimage data. However, in additional or alternative examples, the firstdetector 802 can use any other type of machine learned model fordetecting low variance region(s) associated with the image data. Any ofthe aforementioned machine learned models can be used by the firstdetector 802 for low variance region detection. The first detector 802can generate an output based at least in part on analyzing the imagedata with the machine learned model.

For the purpose of this discussion, the use of the term “output” canrefer to information generated by analyzing the image data using thefirst machine learned model. In some examples, the term “output” canrefer to information that is explicitly modeled. However, in otherexamples, the term “output” can refer to any indication and/orinformation generated by analyzing the image data, including but notlimited to an intermediary output associated with individual layers ofthe first machine learned model. In some examples, an “output” can beassociated with an indication of a low variance region, which can be inthe image data or associated with the image data. In examples where theindication of the low variance region is associated with the image data,such an indication can be derived from the image data and/or can beoutput via some portion of the first machine learned model.

At operation 906, the perception system 322 determines whether a lowvariance region is detected in association with the image data. In atleast one example, the perception system 322 can analyze the output ofthe first detector 802 to determine whether a low variance region isdetected in association with the image data. If a low variance region isnot detected, the first detector 802 can associate information with theimage data indicating that a low variance region was not detected, asillustrated at operation 908. However, if a low variance region isdetected, the first detector 802 can associate an indication with aregion of interest of the image data corresponding to the low varianceregion, as illustrated at operation 910.

As described above, in at least one example, the first detector 802 cangenerate an output, which can include the image data associated with anindication of whether a low variance region was detected by the firstdetector 802. In some examples, the indication can be a bounding boxthat surrounds a region of interest corresponding to low variance regionin the image data. In other examples, any other type of indication canbe used to indicate whether a low variance region was detected by thefirst detector 802. In at least one example, the indication(s) can beassociated with information including an x-position (global and/or localposition), a y-position (global and/or local position), a z-position(global and/or local position), an orientation (e.g., a roll, pitch,yaw), an extent of the object (e.g., height (h), width (w), etc.), anindication of whether a low variance region was detected (e.g., a binaryindication (i.e., low variance region detected (H) or not (N)), aprobability, etc.), etc., as well as any uncertainties associatedtherewith. In at least one example, the perception system 322 cananalyze the output to determine whether the image data is associatedwith indication(s) and/or information indicative that a low varianceregion was detected.

In some examples, the first detector 802 may not associate an indicationsuch as a bounding box with a region of interest and instead, mayprovide an indication of the low variance region (e.g., features,information associated therewith) to a second detector 812, which isshown by the dashed line between operation 906 and 912.

At operation 912, the perception system 322 determines whether a highvariance region is detected in the image data. In at least one example,the second detector 812 can determine whether a high variance region ispresent in the image data. That is, the second detector 812 can analyzethe image data to determine whether a high variance region is present inthe image data. In some examples, the second detector 812 can use one ormore statistical models to determine whether a high variance region ispresent in an environment. For example, statistical priors, derived fromtraining image data, can be used to infer that a high variance region ispresent in image data based on the detection of the low variance region.In some examples, the high variance region can have been previouslycoupled with the low variance region.

As a non-limiting example, if statistical priors indicate that a pair ofheadlights are ⅓ the height of a vehicle and positioned 48 inches apart,a high variance region (e.g., a vehicle) can be inferred to be presentin the image data and an indication can be associated therewith, asdescribed below. In at least some examples, such statistical analysismay be performed based on, for example, binning sizes and/or proportionsof detected low variance regions (e.g., a first bin from 0 to 1, asecond bin from 1 to 1.5, etc. with respect to a height to width ratioof the detected face) and associating a bounding box size and/orrelative position which each bin for the corresponding ratio based onaverages of detections associated therewith. In other examples, variousother statistical analyses may be performed on the data to associate abounding box size and/or position based on the low variance regiondetection. As such, the second detector 812 can detect a high varianceregion. In an additional or alternative example, the second detector 812can use one or more machine learned models to determine whether a highvariance region is present in an environment. Additional details aredescribed below with reference to FIGS. 10-12.

Based at least in part on determining that a high variance region ispresent in the image data, the second detector 812 can associate anindication (e.g., a bounding box) with a region of interest of the imagedata corresponding to a high variance region, as illustrated atoperation 914. As described above, in at least one example, the outputof the second detector 812 can include one or more indications that areassociated with regions of interest corresponding to an identified highvariance region in the environment within which the vehicle 302 ispositioned. In some examples, the indication can be a bounding box thatsurrounds a region of interest corresponding to high variance region inthe image data. In other examples, any other type of indication can beused to indicate that a high variance region was detected in theenvironment by the second detector 812. In at least one example, theindication can be associated with information including an x-position(global and/or local position), a y-position (global and/or localposition), a z-position (global and/or local position), an orientation(e.g., a roll, pitch, yaw), an object type (e.g., a classification), avelocity of the object, an acceleration of the object, an extent of theobject (e.g., height (h), width (w), etc.), etc., as well as anyuncertainties associated therewith.

At operation 916, the perception system 322 can provide the image datato another system of the vehicle 302. In at least one example, theoutput of the perception system 322 can be provided to other vehiclesystem(s) of the vehicle 302. Such system(s) can include a predictionsystem 324, a planning system 326, and so on. In at least one example,the output can be provided to the other vehicle system(s) for makingdeterminations with respect to controlling the vehicle 302, forinstance, for determining a trajectory along which the vehicle 302 is todrive and causing the vehicle 302 to drive along the trajectory.

The output of the perception system 322 can be provided to other vehiclesystem(s) of the vehicle 302 regardless of whether a high varianceregion is detected in the image data. However, if a high variance regionis detected, the output can include image data associated withindication(s) and/or information associated with the detected highvariance region. The other system(s) of the vehicle 302 can use theinput to perform operations as described above with reference to FIG. 3.

FIG. 4 illustrates an example process 400 for using head detectiontechniques for improving pedestrian detection, as described herein.

At operation 402, the perception system 322 receives image data. Asdescribed above, a vehicle 302 can include one or more sensor systems306. The sensor system(s) 306 can include LIDAR sensors, RADAR sensors,ultrasonic transducers, SONAR sensors, location sensors (e.g., GPS,compass, etc.), inertial sensors (e.g., inertial measurement units,accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB,IR, intensity, depth, etc.), wheel encoders, microphones, environmentsensors (e.g., temperature sensors, humidity sensors, light sensors,pressure sensors, etc.), ToF sensors, etc. The sensor system(s) 306 canprovide input to the vehicle computing device(s) 304. In at least oneexample, the sensor system(s) 306 can provide image data to theperception system 322.

At operation 404, the perception system 322 analyzes the image datausing a first machine learned model trained to detect heads. Asdescribed above with reference to FIG. 3, the perception system 322 canperform object detection, segmentation, and/or classification based atleast in part on sensor data received from the sensor system(s) 306. Inat least one example, the perception system 322 can receive the imagedata and can utilize one or more image processing algorithms to performobject detection, segmentation, and/or classification with respect toobject(s) identified in the image data. In at least one example, theperception system 322 can perform head detection using a head detector102. As described above, in at least one example, the head detector 102can utilize a neural network for detecting head(s) in the image dataand/or otherwise associated therewith. However, in additional oralternative examples, the head detector 102 can use any other type ofmachine learned model for detecting head(s) in the image data and/orotherwise associated therewith. Examples of machine learned models thatcan be used by the head detector 102 include, but are not limited to,FaceMatch, Face Everything, Fisherfaces, DeepID Test, Tensorflow,OpenFace, TinyFaces, etc. Any of the aforementioned machine learnedmodels can be used by the head detector 102 for head detection, or amachine learned model trained by the training system 340 can be used bythe head detector 102 for head detection. The head detector 102 cangenerate an output based at least in part on analyzing the image datawith the machine learned model.

At operation 406, the perception system 322 determines whether a head isdetected in the image data. In at least one example, the perceptionsystem 322 can analyze the output of the head detector 102 to determinewhether a head is detected in the image data. If a head is not detectedin the image data, the head detector 102 can associate information withthe image data indicating that a head was not detected, as illustratedat operation 408. However, if a head is detected, the head detector 102can associate an indication (e.g., a bounding box) with a region ofinterest of the image data corresponding to the head, as illustrated atoperation 410.

As described above, in at least one example, the head detector 102 cangenerate an output, which can include the image data associated with anindication of whether a head was detected by the head detector 102. Insome examples, the indication can be a bounding box that surrounds aregion of interest corresponding to a head in the image data. In otherexamples, any other type of indication can be used to indicate whether ahead was detected by the head detector 102. In at least one example, theindication(s) can be associated with information including an x-position(global and/or local position), a y-position (global and/or localposition), a z-position (global and/or local position), an orientation(e.g., a roll, pitch, yaw), an extent of the object (e.g., height (h),width (w), etc.), an indication of whether a head was detected (e.g., abinary indication (i.e., head detected (H) or not (N)), a probability,etc.), etc., as well as any uncertainties associated therewith. In atleast one example, the perception system 322 can analyze the output todetermine whether the image data is associated with indication(s) and/orinformation indicative that a head was detected.

At operation 412, the perception system 322 determines whether apedestrian is detected in the image data. In at least one example, thepedestrian detector 112 can determine whether a pedestrian is present inthe image data. That is, the pedestrian detector 112 can analyze theimage data to determine whether a pedestrian is present in the imagedata. In some examples, the pedestrian detector 112 can use one or morestatistical models to determine whether a pedestrian is present in anenvironment. For example, statistical priors, derived from trainingimage data, can be used to infer that a pedestrian is present in imagedata based on the detection of the head. As a non-limiting example, ifstatistical priors indicate that a head is ⅛ the size of a body and thehead is looking in a particular direction, a pedestrian can be inferredto be present in the image data and an indication can be associatedtherewith, as described below. In at least some examples, suchstatistical analysis may be performed based on, for example, binningsizes and/or proportions of detected heads/faces (e.g., a first bin from0 to 1, a second bin from 1 to 1.5, etc. with respect to a height towidth ratio of the detected face) and associating a bounding box sizeand/or relative position which each bin for the corresponding ratiobased on averages of detections associated therewith. In other examples,various other statistical analyses may be performed on the data toassociate a bounding box size and/or position based on the head/facedetection. As such, the pedestrian detector 112 can detect a pedestrian.In an additional or alternative example, the pedestrian detector 112 canuse one or more machine learned models to determine whether a pedestrianis present in an environment. Additional details are described belowwith reference to FIGS. 5-7.

Based at least in part on determining that a pedestrian is present inthe image data, the pedestrian detector 112 can associate an indication(e.g., a bounding box) with a region of interest of the image datacorresponding to a pedestrian, as illustrated at operation 414. Asdescribed above, in at least one example, the output of the pedestriandetector 112 can include one or more indications that are associatedwith regions of interest corresponding to an identified pedestrian inthe environment within which the vehicle 302 is positioned. In someexamples, the indication can be a bounding box that surrounds a regionof interest corresponding to a pedestrian in the image data. In otherexamples, any other type of indication can be used to indicate that apedestrian was detected in the environment by the pedestrian detector112. In at least one example, the indication can be associated withinformation including an x-position (global and/or local position), ay-position (global and/or local position), a z-position (global and/orlocal position), an orientation (e.g., a roll, pitch, yaw), an objecttype (e.g., a classification), a velocity of the object, an accelerationof the object, an extent of the object (e.g., height (h), width (w),etc.), etc., as well as any uncertainties associated therewith.

At operation 416, the perception system 322 can provide the image datato another system of the vehicle 302. In at least one example, theoutput of the perception system 322 can be provided to other vehiclesystem(s) of the vehicle 302. Such system(s) can include a predictionsystem 324, a planning system 326, and so on. In at least one example,the output can be provided to the other vehicle system(s) for makingdeterminations with respect to controlling the vehicle 302, forinstance, for determining a trajectory along which the vehicle 302 is todrive and causing the vehicle 302 to drive along the trajectory.

The output of the perception system 322 can be provided to other vehiclesystem(s) of the vehicle 302 regardless of whether a pedestrian isdetected in the image data. However, if a pedestrian is detected, theoutput can include image data associated with indication(s) and/orinformation associated with the detected pedestrian. The other system(s)of the vehicle 302 can use the input to perform operations as describedabove with reference to FIG. 3.

FIG. 10 illustrates an example process 1000 for determining whether ahigh variance region is present in image data associated with anenvironment using machine learned models, as described herein.

At operation 1002, the perception system 322 receives image data, asdescribed above at operation 902.

At operation 1004, the perception system 322 analyzes the image datausing a first machine learned model, which can be trained to detect lowvariance regions, to generate a first output. As described above withreference to FIG. 3 and also at operation 904 of FIG. 9, the perceptionsystem 322 can perform object detection, segmentation, and/orclassification based at least in part on sensor data received from thesensor system(s) 306. In at least one example, the perception system 322can receive the image data and can utilize one or more image processingalgorithms to perform object detection, segmentation, and/orclassification with respect to object(s) identified in the image data.In at least one example, the perception system 322 can perform lowvariance region detection using a first detector 802, as described abovewith reference to FIG. 8. As described above, in at least one example,the first detector 802 can utilize a neural network for detecting lowvariance region(s) associated with the image data.

In at least one example, the first detector 802 can generate an output(e.g., a “first output” as used herein FIG. 10), which can include theimage data associated with an indication of whether a low varianceregion was detected by the first detector 802. In some examples, theindication can be a bounding box that surrounds a region of interestcorresponding to a low variance region associated with the image data.In other examples, any other type of indication can be used to indicatewhether a low variance region was detected by the first detector 802. Inat least one example, the indication(s) can be associated withinformation including an x-position (global and/or local position), ay-position (global and/or local position), a z-position (global and/orlocal position), an orientation (e.g., a roll, pitch, yaw), an extent ofthe object (e.g., height (h), width (w), etc.), an indication of whethera low variance region was detected (e.g., a binary indication (i.e., lowvariance region detected (H) or not (N)), a probability, etc.), etc. Inat least one example, the perception system 322 can analyze the outputto determine whether the image data is associated with indication(s)and/or information indicative that a low variance region was detected.

As described above, for the purpose of this discussion, the use of theterm “first output” can refer to information generated by analyzing theimage data using the first machine learned model. In some examples, theterm “first output” can refer to information that is explicitly modeled.However, in other examples, the term “first output” can refer to anyindication and/or information generated by analyzing the image data,including but not limited to an intermediary output associated withindividual layers of the first machine learned model. In some examples,a “first output” can be associated with an indication of a low varianceregion, which can be in the image data or associated with the imagedata. In examples where the indication of the low variance region isassociated with the image data, such an indication can be derived fromthe image data and/or can be output via some portion of the firstmachine learned model.

At operation 1006, the perception system 322 can analyze the firstoutput using a second machine learned model, which can be trained todetect high variance regions, to generate a second output. In at leastone example, the first output can be input into a second machine learnedmodel for detecting high variance regions. The second machine learnedmodel can be trained by the training system 340 as described above withreference to FIG. 3. As described above, in some examples, low varianceand high variance correspondence can be explicit (e.g., by pre-informingmachine learned model(s) of such correspondence). In some examples,machine learned model(s) can be trained to detect low variance features,which can serve as proxies for high variance regions, without explicitlymodelling low variance outputs. Where described in some examples hereinas a low variance detection (or region) informing a high variancedetection, such examples are meant to be inclusive of multiple networks(e.g., in which a low variance network output is used to provideadditional context to a high variance detector network), single networkstrained to determine regions of low variance (e.g., having lossesassociated therewith which are based at least in part on variance in theinput region) and couple to regions of general detections (e.g., bycoupling detection losses with variance-based losses), and/or anycombination thereof, etc.

In such an example, the second detector 812 can analyze the first outputto generate a second output. The second output can include one or moreindications that are associated with a region of interest correspondingto an identified high variance region in the environment within whichthe vehicle 302 is positioned. In some examples, the indication can be abounding box that surrounds a region of interest corresponding to highvariance region in the image data. In other examples, any other type ofindication can be used to indicate that a high variance region wasdetected in the environment by the second detector 812. In at least oneexample, the indication can be associated with information including anx-position (global and/or local position), a y-position (global and/orlocal position), a z-position (global and/or local position), anorientation (e.g., a roll, pitch, yaw), an object type (e.g., aclassification), a velocity of the object, an acceleration of theobject, an extent of the object (e.g., height (h), width (w), etc.),etc.

At operation 1008, the perception system 322 can analyze the secondoutput to determine whether a high variance region is present in theimage data. That is, in at least one example, the perception system 322can analyze the second output to determine whether the image data isassociated with indication(s) and/or information indicative that a highvariance region was detected. The perception system 322 can provide theimage data to other system(s) of the vehicle 302 for controllingoperation of the vehicle 302, as described above with reference to FIGS.3 and 9.

Of course, though depicted in FIG. 10 as distinct operations, operations1004, 1006, and/or 1008 may be combined into a single machine learnedmodel (or network) whereby features are shared between the subnetworks,wherein an output of one is input into another, whereby the outputscomprise separate low variance regions of a network, or the like.

FIG. 5 illustrates an example process 500 for determining whether apedestrian is present in image data associated with an environment usingmachine learned models, as described herein.

At operation 502, the perception system 322 receives image data, asdescribed above at operation 402.

At operation 504, the perception system 322 analyzes the image datausing a first machine learned model, which can be trained to detectheads, to generate a first output. As described above with reference toFIG. 3 and also at operation 404 of FIG. 4, the perception system 322can perform object detection, segmentation, and/or classification basedat least in part on sensor data received from the sensor system(s) 306.In at least one example, the perception system 322 can receive the imagedata and can utilize one or more image processing algorithms to performobject detection, segmentation, and/or classification with respect toobject(s) identified in the image data. In at least one example, theperception system 322 can perform head detection using a head detector102. As described above, in at least one example, the head detector 102can utilize a neural network for detecting head(s) in the image data.

In at least one example, the head detector 102 can generate an output(e.g., a “first output” as used herein FIG. 5), which can include theimage data associated with an indication of whether a head was detectedby the head detector 102. In some examples, the indication can be abounding box that surrounds a region of interest corresponding to a headin the image data. In other examples, any other type of indication canbe used to indicate whether a head was detected by the head detector102. In at least one example, the indication(s) can be associated withinformation including an x-position (global and/or local position), ay-position (global and/or local position), a z-position (global and/orlocal position), an orientation (e.g., a roll, pitch, yaw), an extent ofthe object (e.g., height (h), width (w), etc.), an indication of whethera head was detected (e.g., a binary indication (i.e., head detected (H)or not (N)), a probability, etc.), etc. In at least one example, theperception system 322 can analyze the output to determine whether theimage data is associated with indication(s) and/or informationindicative that a head was detected.

At operation 506, the perception system 322 can analyze the first outputusing a second machine learned model, which can be trained to detectpedestrians, to generate a second output. In at least one example, thefirst output can be input into a machine learned model for detectingpedestrians. The machine learned model can be trained by the trainingsystem 340 as described above with reference to FIG. 3. In such anexample, the pedestrian detector 112 can analyze the first output togenerate a second output. The second output can include one or moreindications that are associated with regions of interest correspondingto an identified pedestrian in the environment within which the vehicle302 is positioned. In some examples, the indication can be a boundingbox that surrounds a region of interest corresponding to pedestrian inthe image data. In other examples, any other type of indication can beused to indicate that a pedestrian was detected in the environment bythe pedestrian detector 112. In at least one example, the indication canbe associated with information including an x-position (global and/orlocal position), a y-position (global and/or local position), az-position (global and/or local position), an orientation (e.g., a roll,pitch, yaw), an object type (e.g., a classification), a velocity of theobject, an acceleration of the object, an extent of the object (e.g.,height (h), width (w), etc.), etc.

At operation 508, the perception system 322 can analyze the secondoutput to determine whether a pedestrian is present in the image data.That is, in at least one example, the perception system 322 can analyzethe second output to determine whether the image data is associated withindication(s) and/or information indicative that a pedestrian wasdetected. The perception system 322 can provide the image data to othersystem(s) of the vehicle 302 for controlling operation of the vehicle302, as described above with reference to FIGS. 3 and 4.

Of course, though depicted in FIG. 5 as distinct operations, operations504, 506, and/or 508 may be combined into a single machine learned model(or network) whereby features are shared between the subnetworks,wherein an output of one is input into another, whereby the outputscomprise separate heads of a network, or the like.

FIG. 11 illustrates another example process 1100 for determining whethera high variance region is present in image data associated with anenvironment using machine learned models, as described herein.

At operation 1102, the perception system 322 receives image data, asdescribed above at operation 902.

At operation 1104, the perception system 322 analyzes the image datausing a first machine learned model trained to detect low varianceregions to generate a first output. As described above with reference toFIG. 3 and also above at operation 904 of FIG. 9, the perception system322 can perform object detection, segmentation, and/or classificationbased at least in part on sensor data received from the sensor system(s)306. In at least one example, the perception system 322 can receive theimage data and can utilize one or more image processing algorithms toperform object detection, segmentation, and/or classification withrespect to object(s) identified in the image data. In at least oneexample, the perception system 322 can perform low variance regiondetection using a first detector 802, as described above with referenceto FIG. 8. As described above, in at least one example, the firstdetector 802 can utilize a neural network for detecting low varianceregion(s) associated with the image data.

As described above, in at least one example, the first detector 802 cangenerate an output (e.g., a “first output” as used here, in FIG. 11),which can include the image data associated with an indication ofwhether a low variance region was detected by the first detector 802. Insome examples, the indication can be a bounding box that surrounds aregion of interest corresponding to a low variance region in the imagedata. In other examples, any other type of indication can be used toindicate whether a low variance region was detected by the firstdetector 802. In at least one example, the indication(s) can beassociated with information including an x-position (global and/or localposition), a y-position (global and/or local position), a z-position(global and/or local position), an orientation (e.g., a roll, pitch,yaw), an extent of the object (e.g., height (h), width (w), etc.), anindication of whether a low variance region was detected (e.g., a binaryindication (i.e., low variance region detected (H) or not (N)), aprobability, etc.), etc. In at least one example, the perception system322 can analyze the output to determine whether the image data isassociated with indication(s) and/or information indicative that a lowvariance region was detected.

As described above, for the purpose of this discussion, the use of theterm “first output” can refer to information generated by analyzing theimage data using the first machine learned model. In some examples, theterm “first output” can refer to information that is explicitly modeled.However, in other examples, the term “first output” can refer to anyindication and/or information generated by analyzing the image data,including but not limited to an intermediary output associated withindividual layers of the first machine learned model. In some examples,a “first output” can be associated with an indication of a low varianceregion, which can be in the image data or associated with the imagedata. In examples where the indication of the low variance region isassociated with the image data, such an indication can be derived fromthe image data and/or can be output via some portion of the firstmachine learned model.

At operation 1106, the perception system 322 analyzes the image datausing a second machine learned model trained to detect objects togenerate a second output. As described above, the perception system 322can perform object detection, segmentation, and/or classification basedat least in part on sensor data received from the sensor system(s) 306of the vehicle 302. In at least one example, the perception system 322can utilize a machine learned model trained to detect and/or classifyobjects in the image data (e.g., an object detector). Such a machinelearned model can be trained by the training system 340 described abovewith reference to FIG. 3. In at least one example, the perception system322 can output an intermediate output of features detected by the secondmachine learned model that are used to detect and/or classify objects inimage data. For the purpose of the discussion of process 1100, theintermediate output of features can be referred to as the “secondoutput.” In at least one example, the first output and the second outputcan be output at or near the same time. That is, in such an example, thefirst machine learned model and the second machine learned model cananalyze the image data at the same time and/or in parallel.

Similar to what is described above, for the purpose of this discussion,the use of the term “second output” can refer to information generatedby analyzing the image data using the second machine learned model. Insome examples, the term “second output” can refer to information that isexplicitly modeled. However, in other examples, the term “second output”can refer to any indication and/or information generated by analyzingthe image data, including but not limited to an intermediary outputassociated with individual layers of the second machine learned model.In some examples, a “second output” can be associated with an indicationof an object (which, in some examples, can be associated with a highvariance region), which can be in the image data or associated with theimage data. In examples where the indication of the object is associatedwith the image data, such an indication can be derived from the imagedata and/or can be output via some portion of the second machine learnedmodel.

At operation 1108, the perception system 322 can analyze the firstoutput and the second output using a third machine learned model trainedto detect high variance regions to generate a third output. In at leastone example, the first output and the second output can be input into amachine learned model for detecting high variance regions, which can beutilized by the second detector 812. In such an example, the seconddetector 812 can analyze the first output and the second output togenerate a third output. The third output can include one or moreindications that are associated with region(s) of interest correspondingto identified high variance region(s) in the environment within whichthe vehicle 302 is positioned. In some examples, an indication can be abounding box that surrounds a region of interest corresponding to a highvariance region in the image data. In other examples, any other type ofindication can be used to indicate that a high variance region wasdetected in the environment by the second detector 812. In at least oneexample, the indication can be associated with information including anx-position (global and/or local position), a y-position (global and/orlocal position), a z-position (global and/or local position), anorientation (e.g., a roll, pitch, yaw), an object type (e.g., aclassification), a velocity of the object, an acceleration of theobject, an extent of the object (e.g., height (h), width (w), etc.),etc.

As described above, in some examples, low variance and high variancecorrespondence can be explicit (e.g., by pre-informing machine learnedmodel(s) of such correspondence). In some examples, machine learnedmodel(s) can be trained to detect low variance features, which can serveas proxies for high variance regions, without explicitly modelling lowvariance outputs. Where described in some examples herein as a lowvariance detection (or region) informing a high variance detection, suchexamples are meant to be inclusive of multiple networks (e.g., in whicha low variance network output is used to provide additional context to ahigh variance detector network), single networks trained to determineregions of low variance (e.g., having losses associated therewith whichare based at least in part on variance in the input region) and coupleto regions of general detections (e.g., by coupling detection losseswith variance-based losses), and/or any combination thereof, etc.

At operation 1110, the perception system 322 can analyze the thirdoutput to determine whether a high variance region is present in theimage data. That is, in at least one example, the perception system 322can analyze the third output to determine whether the image data isassociated with indication(s) and/or information indicative that a highvariance region was detected. The perception system 322 can provide theimage data to other system(s) of the vehicle 302 for controllingoperation of the vehicle 302, as described above with reference to FIGS.3 and 9.

Of course, though depicted in FIG. 11 as distinct operations, operations1104, 1106, and/or 1108 may be combined into a single machine learnedmodel (or network) whereby features are shared between the subnetworks,wherein an output of one is input into another, whereby the outputscomprise separate low variance regions of a network, or the like.

FIG. 6 illustrates another example process 600 for determining whether apedestrian is present in image data associated with an environment usingmachine learned models, as described herein.

At operation 602, the perception system 322 receives image data, asdescribed above at operation 402.

At operation 604, the perception system 322 analyzes the image datausing a first machine learned model trained to detect heads to generatea first output. As described above with reference to FIG. 3 and alsoabove at operation 404 of FIG. 4, the perception system 322 can performobject detection, segmentation, and/or classification based at least inpart on sensor data received from the sensor system(s) 306. In at leastone example, the perception system 322 can receive the image data andcan utilize one or more image processing algorithms to perform objectdetection, segmentation, and/or classification with respect to object(s)identified in the image data. In at least one example, the perceptionsystem 322 can perform head detection using a head detector 102. Asdescribed above, in at least one example, the head detector 102 canutilize a neural network for detecting head(s) in the image data.

As described above, in at least one example, the head detector 102 cangenerate an output (e.g., a “first output” as used here, in FIG. 6),which can include the image data associated with an indication ofwhether a head was detected by the head detector 102. In some examples,the indication can be a bounding box that surrounds a region of interestcorresponding to a head in the image data. In other examples, any othertype of indication can be used to indicate whether a head was detectedby the head detector 102. In at least one example, the indication(s) canbe associated with information including an x-position (global and/orlocal position), a y-position (global and/or local position), az-position (global and/or local position), an orientation (e.g., a roll,pitch, yaw), an extent of the object (e.g., height (h), width (w),etc.), an indication of whether a head was detected (e.g., a binaryindication (i.e., head detected (H) or not (N)), a probability, etc.),etc. In at least one example, the perception system 322 can analyze theoutput to determine whether the image data is associated withindication(s) and/or information indicative that a head was detected.

At operation 606, the perception system 322 analyzes the image datausing a second machine learned model trained to detect objects togenerate a second output. As described above, the perception system 322can perform object detection, segmentation, and/or classification basedat least in part on sensor data received from the sensor system(s) 306of the vehicle 302. In at least one example, the perception system 322can utilize a machine learned model trained to detect and/or classifyobjects in the image data (e.g., an object detector). Such a machinelearned model can be trained by the training system 340 described abovewith reference to FIG. 3. In at least one example, the perception system322 can output an intermediate output of features detected by themachine learned model that are used to detect and/or classify objects inimage data. For the purpose of the discussion process 600, theintermediate output of features can be referred to as the “secondoutput.” In at least one example, the first output and the second outputcan be output at or near the same time. That is, in such an example, thefirst machine learned model and the second machine learned model cananalyze the image data at the same time and/or in parallel.

At operation 608, the perception system 322 can analyze the first outputand the second output using a third machine learned model trained todetect pedestrians to generate a third output. In at least one example,the first output and the second output can be input into a machinelearned model for detecting pedestrians, which can be utilized by thepedestrian detector 112. In such an example, the pedestrian detector 112can analyze the first output and the second output to generate a thirdoutput. The third output can include one or more indications that areassociated with regions of interest corresponding to identifiedpedestrians in the environment within which the vehicle 302 ispositioned. In some examples, an indication can be a bounding box thatsurrounds a region of interest corresponding to a pedestrian in theimage data. In other examples, any other type of indication can be usedto indicate that a pedestrian was detected in the environment by thepedestrian detector 112. In at least one example, the indication can beassociated with information including an x-position (global and/or localposition), a y-position (global and/or local position), a z-position(global and/or local position), an orientation (e.g., a roll, pitch,yaw), an object type (e.g., a classification), a velocity of the object,an acceleration of the object, an extent of the object (e.g., height(h), width (w), etc.), etc.

At operation 610, the perception system 322 can analyze the third outputto determine whether a pedestrian is present in the image data. That is,in at least one example, the perception system 322 can analyze the thirdoutput to determine whether the image data is associated withindication(s) and/or information indicative that a pedestrian wasdetected. The perception system 322 can provide the image data to othersystem(s) of the vehicle 302 for controlling operation of the vehicle302, as described above with reference to FIGS. 3 and 4.

Of course, though depicted in FIG. 6 as distinct operations, operations604, 606, and/or 608 may be combined into a single machine learned model(or network) whereby features are shared between the subnetworks,wherein an output of one is input into another, whereby the outputscomprise separate heads of a network, or the like.

FIGS. 6 and 11 are examples of processes where the indication of thehigh variance region (e.g., pedestrian) can be determined independent ofan indication of a low variance region (e.g., head). That is, in someexamples, the second detector 812 can determine an indication of a highvariance region (e.g., pedestrian) based on the sensor data, or aportion thereof, and without the indication of the low variance region(e.g., head). In an example, the first machine learned model may notdetect a low variance region (e.g., head) but the second machine learnedmodel may generate an output that can be used by the third machinelearned model to detect an indication of a high variance region (e.g.,pedestrian). In an additional example, the third machine learned modelcan utilize outputs of both the first and second machine learned model,as described above.

FIG. 12 illustrates yet another example process 1200 for determiningwhether a high variance region is present in image data associated withan environment using machine learned models, as described herein.

At operation 1202, the perception system 322 receives image data, asdescribed above at operation 902.

At operation 1204, the perception system 322 analyzes the image datausing a first machine learned model trained to detect low varianceregions to generate a first output. As described above with reference toFIG. 3 and also at operation 904 of FIG. 9, the perception system 322can perform object detection, segmentation, and/or classification basedat least in part on sensor data received from the sensor system(s) 306.In at least one example, the perception system 322 can receive the imagedata and can utilize one or more image processing algorithms to performobject detection, segmentation, and/or classification with respect toobject(s) identified in the image data. In at least one example, theperception system 322 can perform low variance region detection using afirst detector 802, as described above with reference to FIG. 8. Asdescribed above, in at least one example, the first detector 802 canutilize a neural network for detecting low variance region(s) associatedwith the image data.

As described above, in at least one example, the first detector 802 cangenerate an output (e.g., a “first output” as used here, in FIG. 12),which can include the image data associated with an indication ofwhether a low variance region was detected by the first detector 802. Insome examples, the indication can be a bounding box that surrounds aregion of interest corresponding to a low variance region associatedwith the image data. In other examples, any other type of indication canbe used to indicate whether a low variance region was detected by thefirst detector 802. In at least one example, the indication(s) can beassociated with information including an x-position (global and/or localposition), a y-position (global and/or local position), a z-position(global and/or local position), an orientation (e.g., a roll, pitch,yaw), an extent of the object (e.g., height (h), width (w), etc.), anindication of whether a low variance region was detected (e.g., a binaryindication (i.e., low variance region detected (H) or not (N)), aprobability, etc.), etc. In at least one example, the perception system322 can analyze the output to determine whether the image data isassociated with indication(s) and/or information indicative that a lowvariance region was detected.

As described above, for the purpose of this discussion, the use of theterm “first output” can refer to information generated by analyzing theimage data using the first machine learned model. In some examples, theterm “first output” can refer to information that is explicitly modeled.However, in other examples, the term “first output” can refer to anyindication and/or information generated by analyzing the image data,including but not limited to an intermediary output associated withindividual layers of the first machine learned model. In some examples,a “first output” can be associated with an indication of a low varianceregion, which can be in the image data or associated with the imagedata. In examples where the indication of the low variance region isassociated with the image data, such an indication can be derived fromthe image data and/or can be output via some portion of the firstmachine learned model.

At operation 1206, the perception system 322 analyzes the image datausing a second machine learned model trained to detect objects togenerate a second output, the second output associated with a pluralityof classifications. As described above with reference to FIG. 3 and alsoat operation 1106 of FIG. 11, the perception system 322 can performobject detection, segmentation, and/or classification based at least inpart on sensor data received from the sensor system(s) 306 of thevehicle 302. In at least one example, the perception system 322 canutilize a machine learned model trained to detect and/or classifyobjects in the image data. Such a machine learned model can be trainedby the training system 340 described above with reference to FIG. 3. Inat least one example, for an object identified in the image data, themachine learned model can output a plurality of classificationsassociated with the object. For example, the machine learned model canbe a multi-class classifier that is trained, via a machine learningmechanism, to output a probability that an object is associated withindividual classes of the multi-class classifier. In some examples, theclassifications can be ranked from highest probability to lowestprobability, and systems of the vehicle 302 can use the rankedclassifications for determining (i) an object type associated with theobject and/or (ii) how to control the vehicle 302. For the purpose ofthe discussion of process 1200, the plurality of classifications can bereferred to as the “second output.”

In at least one example, the first output and the second output can beoutput at or near the same time. That is, in such an example, the firstmachine learned model and the second machine learned model can analyzethe image data at the same time and/or in parallel.

At operation 1208, the perception system 322 determines whether a lowvariance region is detected in association with the image data. In atleast one example, the perception system 322 can analyze the firstoutput of the first machine learned model to determine whether a lowvariance region is detected in association with the image data. Asdescribed above with reference to operation 406 of FIG. 4, if a lowvariance region is not detected, the first detector 802 can associateinformation with the image data indicating that a low variance regionwas not detected. However, if a low variance region is detected, thefirst detector 802 can associate an indication (e.g., a bounding box)with a region of interest of the image data corresponding to the lowvariance region. In at least one example, the perception system 322 cananalyze the first output to determine whether the image data isassociated with indication(s) and/or information indicative that a lowvariance region was detected.

At operation 1210, the perception system 322 reduces a thresholdassociated with a classification of the plurality of classifications. Inat least one example, each classification of the plurality ofclassifications can be associated with a threshold that is configurable.In at least one example, the threshold can be set to a value such thatif a probability (certainty or, inversely, an uncertainty) associatedwith a classification meets or exceeds the threshold, the perceptionsystem 322 can determine that an object is associated with theclassification. In at least one example, responsive to determining thata low variance region was detected in the image data, the perceptionsystem 322 can reduce a threshold associated with a high variance regionclassification associated with the low variance region. That is, if alow variance region is detected in the image data, it is more likelythat an object associated with a high variance region that is associatedwith the low variance region than if a low variance region is notdetected in the image data. As such, the threshold associated with aclassification indicating that an object is a high variance region neednot be as high as when low variance region detection techniques are notused. As such, the perception system 322 can reduce the thresholdassociated with a high variance region classification.

At operation 1212, the perception system 322 determines whether aprobability associated with a classification (e.g., high varianceregion) meets or exceeds a threshold. In at least one example, theperception system 322 can compare the probability associated with thehigh variance region classification to the threshold associated with thehigh variance region classification to determine whether the probabilitymeets or exceeds the threshold. Based at least in part on determiningthat the probability meets or exceeds the threshold, the perceptionsystem 322 can determine that a high variance region is present in theimage data, as illustrated at operation 1214. Based at least in part ondetermining that the probability does not meet or exceed the threshold,the perception system 322 can determine that a high variance region isnot present in the image data, as illustrated at operation 1216. Theperception system 322 can provide the image data to other system(s) ofthe vehicle 302 for controlling operation of the vehicle 302, asdescribed above with reference to FIGS. 3 and 4.

FIG. 7 illustrates yet another example process 700 for determiningwhether a pedestrian is present in image data associated with anenvironment using machine learned models, as described herein.

At operation 702, the perception system 322 receives image data, asdescribed above at operation 402.

At operation 704, the perception system 322 analyzes the image datausing a first machine learned model trained to detect heads to generatea first output. As described above with reference to FIG. 3 and also atoperation 404 of FIG. 4, the perception system 322 can perform objectdetection, segmentation, and/or classification based at least in part onsensor data received from the sensor system(s) 306. In at least oneexample, the perception system 322 can receive the image data and canutilize one or more image processing algorithms to perform objectdetection, segmentation, and/or classification with respect to object(s)identified in the image data. In at least one example, the perceptionsystem 322 can perform head detection using a head detector 102. Asdescribed above, in at least one example, the head detector 102 canutilize a neural network for detecting head(s) in the image data.

As described above, in at least one example, the head detector 102 cangenerate an output (e.g., a “first output” as used here, in FIG. 7),which can include the image data associated with an indication ofwhether a head was detected by the head detector 102. In some examples,the indication can be a bounding box that surrounds a region of interestcorresponding to a head in the image data. In other examples, any othertype of indication can be used to indicate whether a head was detectedby the head detector 102. In at least one example, the indication(s) canbe associated with information including an x-position (global and/orlocal position), a y-position (global and/or local position), az-position (global and/or local position), an orientation (e.g., a roll,pitch, yaw), an extent of the object (e.g., height (h), width (w),etc.), an indication of whether a head was detected (e.g., a binaryindication (i.e., head detected (H) or not (N)), a probability, etc.),etc. In at least one example, the perception system 322 can analyze theoutput to determine whether the image data is associated withindication(s) and/or information indicative that a head was detected.

At operation 706, the perception system 322 analyzes the image datausing a second machine learned model trained to detect objects togenerate a second output, the second output associated with a pluralityof classifications. As described above with reference to FIG. 3 and alsoat operation 606 of FIG. 6, the perception system 322 can perform objectdetection, segmentation, and/or classification based at least in part onsensor data received from the sensor system(s) 306 of the vehicle 302.In at least one example, the perception system 322 can utilize a machinelearned model trained to detect and/or classify objects in the imagedata. Such a machine learned model can be trained by the training system340 described above with reference to FIG. 3. In at least one example,for an object identified in the image data, the machine learned modelcan output a plurality of classifications associated with the object.For example, the machine learned model can be a multi-class classifierthat is trained, via a machine learning mechanism, to output aprobability that an object is associated with individual classes of themulti-class classifier. In some examples, the classifications can beranked from highest probability to lowest probability, and systems ofthe vehicle 302 can use the ranked classifications for determining (i)an object type associated with the object and/or (ii) how to control thevehicle 302. For the purpose of the discussion of process 700, theplurality of classifications can be referred to as the “second output.”In at least one example, the first output and the second output can beoutput at or near the same time. That is, in such an example, the firstmachine learned model and the second machine learned model can analyzethe image data at the same time and/or in parallel.

At operation 708, the perception system 322 determines whether a head isdetected in the image data. In at least one example, the perceptionsystem 322 can analyze the first output of the first machine learnedmodel to determine whether a head is detected in the image data. Asdescribed above with reference to operation 406 of FIG. 4, if a head isnot detected in the image data, the head detector 102 can associateinformation with the image data indicating that a head was not detected.However, if a head is detected, the head detector 102 can associate anindication (e.g., a bounding box) with a region of interest of the imagedata corresponding to the head. In at least one example, the perceptionsystem 322 can analyze the first output to determine whether the imagedata is associated with indication(s) and/or information indicative thata head was detected.

At operation 710, the perception system 322 reduces a thresholdassociated with a classification of the plurality of classifications. Inat least one example, each classification of the plurality ofclassifications can be associated with a threshold that is configurable.In at least one example, the threshold can be set to a value such thatif a probability (certainty or, inversely, an uncertainty) associatedwith a classification meets or exceeds the threshold, the perceptionsystem 322 can determine that an object is associated with theclassification. In at least one example, responsive to determining thata head was detected in the image data, the perception system 322 canreduce a threshold associated with a pedestrian classification. That is,if a head is detected in the image data, it is more likely that anobject is a pedestrian than if a head is not detected in the image data.As such, the threshold associated with a classification indicating thatan object is a pedestrian need not be as high as when head detectiontechniques are not used. As such, the perception system 322 can reducethe threshold associated with a pedestrian classification.

At operation 712, the perception system 322 determines whether aprobability associated with a classification (e.g., pedestrian) meets orexceeds a threshold. In at least one example, the perception system 322can compare the probability associated with the pedestrianclassification to the threshold associated with the pedestrianclassification to determine whether the probability meets or exceeds thethreshold. Based at least in part on determining that the probabilitymeets or exceeds the threshold, the perception system 322 can determinethat a pedestrian is present in the image data, as illustrated atoperation 714. Based at least in part on determining that theprobability does not meet or exceed the threshold, the perception system322 can determine that a pedestrian is not present in the image data, asillustrated at operation 716. The perception system 322 can provide theimage data to other system(s) of the vehicle 302 for controllingoperation of the vehicle 302, as described above with reference to FIGS.3 and 4.

FIG. 13 illustrates an example process 1300 for training a model todetect high variance region(s) and/or low variance region(s).

Block 1302 illustrates receiving data associated with environments ofvehicles. As described above, in at least one example, the computingdevice(s) 334 can receive the sensor data (raw or processed) from thevehicle 302 and/or other data collection devices (which can includeother vehicles like 302), as well as data from one or more third partysources and/or systems. For example, individual vehicles, such as thevehicle 302, can include sensor systems. As described above withreference to FIG. 3, the sensor system(s) 306 can include LIDAR sensors,RADAR sensors, ultrasonic transducers, sound navigation and ranging(SONAR) sensors, location sensors (e.g., global positioning system(GPS), compass, etc.), inertial sensors (e.g., inertial measurementunits, accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g.,RGB, IR, intensity, depth, etc.), wheel encoders, microphones,environment sensors (e.g., temperature sensors, humidity sensors, lightsensors, pressure sensors, etc.), ToF sensors, etc. The sensor system(s)306 can include multiple instances of each of these or other types ofsensors. In at least one example, the sensor system(s) 306 can sendsensor data, via the network(s) 332, to the computing device(s) 334 at aparticular frequency, after a lapse of a predetermined period of time,in near real-time, etc. In some examples, the data can be image data, abatch of image data, or an image space. As described above, the data canadditionally or alternatively comprise data from other sensors of thesensor system(s) 306.

Block 1304 illustrates determining annotated data based on the data, theannotated data including an annotated high variance region and/or anannotated low variance region. In at least one example, the trainingsystem 340 can annotate regions of the data and/or receive annotateddata. Such data can comprise training data.

In some examples, the training system 340 can annotate regions of thedata using one or more statistical models. As described above, in someexamples, the training data can include images tagged, or otherwiseannotated, to indicate where a high variance region is located. In otherexamples, high variance regions may not be tagged or otherwiseannotated. As described above, a high variance region is a region wheremultiple instances of the region (e.g., regions classified with a samebounding box) in input data (e.g., image data, a batch of image data, aninput space) are compared, a variance metric associated with themultiple instances of the region can meet or exceed a threshold. Such avariance metric can represent information inside of a bounding boxassociated with a region. In some examples, the variance metric can bebased on entropy, pixel intensity, aspect ratios between boxes, and thelike.

As described above, in some examples, the training data can includeimages tagged, or otherwise annotated, to indicate where a low varianceregion is located. As described above, a low variance region is a regionwhere multiple instances of the region (e.g., regions classified with asame bounding box) in input data (e.g., image data, a batch of imagedata, an input space) are compared, a variance metric associated withthe multiple instances of the region is below a threshold. Such avariance metric can represent information inside of a bounding boxassociated with a region. In some examples, the variance metric can bebased on entropy, pixel intensity, aspect ratios between boxes, and thelike.

In some examples, the high variance region and/or low variance regioncan be determined based at least in part on a feature associated withthe data. For example, in some examples, the training system 340 cantrain, using machine learning, a model for detecting high varianceregions, based on low variance features. In such examples, the lowvariance outputs may not be explicitly modeled, but instead, a model canbe trained to detect high variance regions based at least in part onimplicit low variance regions. In an example, input data (e.g., imagedata) can be projected (e.g., via a neural network) into a feature spacethat represents the input data. Such feature space includes high-levelinformation for object detection. That is, when a higher variance regionor input data is be projected into a lower dimensional space, the resultcan be a space having a lower variance. Such a result can be tagged as,or otherwise indicated to be, a low variance region. In an example,another neural network can be used to reconstruct the input data (e.g.,image data) based on features in the feature space. That is, thefeatures can be projected into an image space (e.g., using a neuralnetwork). In at least one example, the training system 340 can performsuch projection and re-projection to determine high variance and/or lowvariance regions.

Block 1306 illustrates determining whether the data is sufficient fortraining. In at least one example, the more training data available fortraining a model, the more accurate the model is likely to be. As such,the training system 340 can determine whether there is enough trainingdata to train the model. In some examples, the training system 340 cancompare the amount of data and/or the quality of data to one or morethresholds and, if the training data satisfies the threshold(s), candetermine that the data is sufficient for training the model. If thedata is not sufficient, process 1300 can return to block 1302.

Block 1308 illustrates inputting the data into a model. In at least oneexample, the training system 340 can input the data into a model. Themodel can determine an output, as illustrated at block 1310, wherein theoutput comprises a high variance output and a low variance output. In atleast one example, the training system 340 can train a model such thatthe trained model is configured to output an indication of a lowvariance region and an indication of a high variance region based atleast in part on an input. In some examples, the training system 340 cantrain multiple models to perform detection operations as describedherein. As described above, machine trained models described herein maycomprise separate networks, may comprise different aspects of a commonnetwork, or may otherwise comprise a combination of multiple networks.In any of the examples provided herein, any of the machine learnedmodels may be trained end-to-end, jointly, disjointly, or the like. Insuch examples where the models are trained end-to-end or jointly, commonfeatures may be leveraged by one network to improve outputs of theother.

In at least one example, the training system 340 can train the machinelearned model(s) (e.g., “model” of FIG. 13) independently and, in someexamples, using pre-informed of relationships between low varianceregions and high variance regions. In some examples, the machine learnedmodel(s) can be trained end-to-end. When trained end-to-end, suchtraining may enforce the coupling between low variance regions whichinform the ultimate output. In some examples, training such a machinelearned model to detect a low variance region may comprise evaluatingand back-propagating based at least in part on a loss function whichdetermines a measure of the variance of an associated region in theinput space.

Block 1312 illustrates determining a difference between the output andthe annotated data. In at least one example, the training system 340 candetermine a difference between the output and the annotated data.

As described above, in some examples, the training system 340 can train,using machine learning, a model for detecting high variance regions,based on low variance features. In some examples, such training can bedone by modelling an end-to-end optimization problem. In such examples,the low variance outputs may not be explicitly modeled, but instead, amodel can be trained to detect high variance regions based at least inpart on implicit low variance regions. In an example, input data (e.g.,image data) can be projected (e.g., via a neural network) into a featurespace that represents the input data. Such feature space includeshigh-level information for object detection. In an example, anotherneural network can be used to reconstruct the input data (e.g., imagedata) based on features in the feature space. That is, the features canbe projected into an image space (e.g., using a neural network). In atleast one example, a loss function can be employed to enforceconsistency between the input data and the reconstructed input data.That is, a loss function can determine a difference between the outputand the annotated data and can be employed by the training system 340 totrain the model to focus on low variance features for detecting highvariance regions (e.g., explicitly modelling high variance regions). Inat least one example, the training system 340 can thus alter parametersof the model based partly on the difference, as illustrated at block1314.

Block 1316 illustrates transmitting the model to a vehicle configured tobe controlled by another output of the model. In at least one example,the trained model can be used by the perception system 322 and/or thedetector(s) (e.g., the first detector 802 and/or the second detector804) to detect high variance regions and/or low variance regions inimage data, as described above. As described above, in some examples, ahigh variance region can correspond to a pedestrian and a low varianceregion can correspond to a head, or portion thereof.

Example Clauses

A. A method comprising: receiving, from an image capturing device on avehicle, image data representing an environment; inputting, into amachine learned model, at least a portion of the image data; receiving,from the machine learned model, an indication of a head represented inthe image data, the indication of the head comprising a first boundingbox; determining, based at least partly on the indication of the head,that a pedestrian is present in the environment; determining, based atleast partly on the indication of the head, an indication of thepedestrian, the indication of the pedestrian comprising a secondbounding box; providing the indication of the pedestrian to at least oneof a planning component or a prediction component of the vehicle fordetermining a trajectory along which the vehicle is to travel; andcontrolling the vehicle based at least partly on the trajectory.

B. The method as paragraph A recites, wherein determining the indicationof the pedestrian comprises: performing a statistical analysis of theindication of the head and an associated portion of the image data thatis proximate to the head; and determining, based at least partly on thestatistical analysis and the associated portion of the image data thatis proximate to the head, a position and extents of the indication ofthe pedestrian.

C. The method as paragraph A or B recites, wherein the machine learnedmodel comprises a first machine learned model and the indication of thehead comprises a first output, and wherein determining the indication ofthe pedestrian comprises: inputting the first output into a secondmachine learned model; receiving, from the second machine learned model,the indication of the pedestrian, wherein the second machine learnedmodel is trained to detect pedestrians in image data.

D. The method as any of claims A-C recites, wherein the machine learnedmodel comprises a first machine learned model and the indication of thehead comprises a first output, and wherein determining the indication ofthe pedestrian comprises: receiving, based at least partly on analyzingthe image data using a second machine learned model, a second outputindicating at least one feature of the pedestrian; inputting the firstoutput and the second output into a third machine learned model, whereinthe third machine learned model is trained to detect pedestrians inimage data; and receiving, from the third machine learned model, theindication of the pedestrian.

E. The method as any of claims A-D recites, wherein the machine learnedmodel comprises a first machine learned model and the indication of thehead comprises a first output, the method further comprising: receiving,based at least partly on analyzing the image data using a second machinelearned model, a second output indicating a plurality of classificationsof an object identified in the image data and a plurality of confidencescores, an individual confidence score corresponding to an individualclassification; reducing, based at least partly on the first output, athreshold associated with a pedestrian classification of the pluralityof classifications; determining that a confidence score associated withthe pedestrian classification meets or exceeds the threshold; anddetermining that the pedestrian is present in the environment based atleast partly on determining that the confidence score meets or exceedsthe threshold.

F. The method as any of claims A-E recites, further comprisingdetermining information associated with the pedestrian, wherein theinformation comprises at least one of an x-position of the pedestrian, ay-position of the pedestrian, a z-position of the pedestrian, anorientation of the pedestrian, or an extent of the pedestrian.

G. A system comprising: one or more processors; and computer-readableinstructions that, when executed by the one or more processors, causethe one or more processors to perform operations comprising: receiving,from a sensor associated with the system, sensor data representing anenvironment; inputting, into a machine learned model, at least a portionof the sensor data; receiving, from the machine learned model, anindication of a head represented in the sensor data; determining anindication of the pedestrian based at least partly on the indication ofthe head; determining, based at least partly on the indication of thehead or the indication of the pedestrian, that a pedestrian is presentin the environment; and providing the indication of the pedestrian to acomponent of the system for controlling the system.

H. The system as paragraph G recites, wherein the machine learned modelis a first machine learned model that outputs a first output includingthe indication of the head and information associated with the head,wherein the information comprises at least one of an x-position of thehead, a y-position of the head, a z-position of the head, an orientationof the head, or an extent of the head.

I. The system as paragraph H recites, wherein determining the indicationof the pedestrian comprises: inputting the first output into a secondmachine learned model; receiving, from the second machine learned model,the indication of the pedestrian, wherein the second machine learnedmodel is trained to detect pedestrians in image data.

J. The system as paragraph H or I recites, wherein determining theindication of the pedestrian comprises: receiving, based at least partlyon analyzing the sensor data using a second machine learned model, asecond output indicating at least one feature of the pedestrian;inputting the first output and the second output into a third machinelearned model, wherein the third machine learned model is trained todetect pedestrians in sensor data; and receiving, from the third machinelearned model, the indication of the pedestrian.

K. The system as any of claims H-J recites, the operations furthercomprising: receiving, based at least partly on analyzing the sensordata using a second machine learned model, a second output indicating aplurality of classifications of an object identified in the sensor dataand a plurality of confidence scores, an individual confidence scorecorresponding to an individual classification; reducing, based at leastpartly on the first output, a threshold associated with a pedestrianclassification of the plurality of classifications; determining that aconfidence score associated with the pedestrian classification meets orexceeds the threshold; and determining that the pedestrian is present inthe environment based at least partly on determining that the confidencescore meets or exceeds the threshold.

L. The system as any of claims G-K recites, wherein the indication ofthe pedestrian is a bounding box that is determined based at leastpartly on one or more of an x-position of the pedestrian, a y-positionof the pedestrian, a z-position of the pedestrian, an orientation of thepedestrian, or an extent of the pedestrian.

M. The system as any of claims G-L recites, wherein the sensor data isimage data and detecting the head comprises detecting at least one of afront view of a face, a side view of a face, or a back of the head inthe image data.

N. The system as any of claims G-M recites, wherein determining theindication of the pedestrian comprises: performing a statisticalanalysis of the indication of the head and an associated portion of thesensor data that is proximate to the head; and determining, based atleast partly on the statistical analysis and the associated portion ofthe sensor data that is proximate to the head, a position and extents ofthe indication of the pedestrian, wherein the indication of thepedestrian is determined based at least partly on the indication of thehead.

O. The system as any of claims G-N recites, wherein the system is avehicle and the operations further comprise: providing the indication ofthe pedestrian to at least one of a planning component or a predictioncomponent for determining a trajectory along which the vehicle is totravel; and controlling the vehicle based at least partly on thetrajectory.

P. One or more computer-readable media that, when executed by one ormore processors, cause the one or more processors to perform operationscomprising: receiving, from a sensor associated with a vehicle, sensordata representing an environment within which the vehicle is located;inputting, into a machine learned model, at least a portion of thesensor data; receiving, from the machine learned model, an indication ofa head represented in the sensor data; determining an indication of thepedestrian based at least partly on the indication of the head;determining, based at least partly on the indication of the head or theindication of the pedestrian, that a pedestrian is present in theenvironment; and providing the indication of the pedestrian to acomponent of the vehicle for controlling the vehicle.

Q. The one or more computer-readable media as paragraph P recites,wherein the machine learned model comprises a first machine learnedmodel and the indication of the head comprises a first output, andwherein determining the indication of the pedestrian comprises:inputting the first output into a second machine learned model;receiving, from the second machine learned model, the indication of thepedestrian, wherein the second machine learned model is trained todetect pedestrians in image data.

R. The one or more computer-readable media as paragraph P or Q recites,wherein the machine learned model comprises a first machine learnedmodel and the indication of the head comprises a first output, andwherein determining the indication of the pedestrian comprises:receiving, based at least partly on analyzing the sensor data using asecond machine learned model, a second output indicating at least onefeature of the pedestrian; inputting the first output and the secondoutput into a third machine learned model, wherein the third machinelearned model is trained to detect pedestrians in sensor data; andreceiving, from the third machine learned model, the indication of thepedestrian.

S. The one or more computer-readable media as any of claims P-R recites,wherein the machine learned model comprises a first machine learnedmodel and the indication of the head comprises a first output, theoperations further comprising: receiving, based at least partly onanalyzing the sensor data using a second machine learned model, a secondoutput indicating a plurality of classifications of an object identifiedin the sensor data and a plurality of confidence scores, an individualconfidence score corresponding to an individual classification;reducing, based at least partly on the first output, a thresholdassociated with a pedestrian classification of the plurality ofclassifications; determining that a confidence score associated with thepedestrian classification meets or exceeds the threshold; anddetermining that the pedestrian is present in the environment based atleast partly on determining that the confidence score meets or exceedsthe threshold.

T. The one or more computer-readable media as any of claims P-S recites,wherein determining the indication of the pedestrian comprises:performing a statistical analysis of the indication of the head and anassociated portion of the sensor data that is proximate to the head; anddetermining, based at least partly on the statistical analysis and theassociated portion of the sensor data that is proximate to the head, aposition and extents of the indication of the pedestrian, wherein theindication of the head is associated with a first bounding box and theindication of the pedestrian is associated with a second bounding box.

U. A method comprising: receiving, from an image capturing device on avehicle, image data representing an environment; inputting, into amachine learned model, at least a portion of the image data; receiving,from the machine learned model, an indication of a low variance regionrepresented in the image data, the indication of the low variance regioncomprising a first bounding box; determining, based at least partly onthe indication of the low variance region, that a high variance regionis present in the environment; determining, based at least partly on theindication of the low variance region, an indication of the highvariance region, the indication of the high variance region comprising asecond bounding box; providing the indication of the high varianceregion to at least one of a planning component or a prediction componentof the vehicle for determining a trajectory along which the vehicle isto travel; and controlling the vehicle based at least partly on thetrajectory.

V. The method as paragraph U recites, wherein determining the indicationof the high variance region comprises: performing a statistical analysisof the indication of the low variance region and an associated portionof the image data that is proximate to the low variance region; anddetermining, based at least partly on the statistical analysis and theassociated portion of the image data that is proximate to the lowvariance region, a position and extents of the indication of the highvariance region.

W. The method as paragraph U or V recites, wherein the machine learnedmodel comprises a first machine learned model and the indication of thelow variance region comprises a first output, and wherein determiningthe indication of the high variance region comprises: inputting thefirst output into a second machine learned model; receiving, from thesecond machine learned model, the indication of the high varianceregion, wherein the second machine learned model is trained to detecthigh variance regions in image data.

X. The method as any of claims U-W recites, wherein the machine learnedmodel comprises a first machine learned model and the indication of thelow variance region comprises a first output, and wherein determiningthe indication of the high variance region comprises: receiving, basedat least partly on analyzing the image data using a second machinelearned model, a second output indicating at least one feature of thehigh variance region; inputting the first output and the second outputinto a third machine learned model, wherein the third machine learnedmodel is trained to detect high variance regions in image data; andreceiving, from the third machine learned model, the indication of thehigh variance region.

Y. The method as any of claims U-X recites, wherein the machine learnedmodel comprises a first machine learned model and the indication of thelow variance region comprises a first output, the method furthercomprising: receiving, based at least partly on analyzing the image datausing a second machine learned model, a second output indicating aplurality of classifications of an object identified in the image dataand a plurality of confidence scores, an individual confidence scorecorresponding to an individual classification; reducing, based at leastpartly on the first output, a threshold associated with a high varianceregion classification of the plurality of classifications; determiningthat a confidence score associated with the high variance regionclassification meets or exceeds the threshold; and determining that thehigh variance region is present in the environment based at least partlyon determining that the confidence score meets or exceeds the threshold.

Z. The method as any of claims U-Y recites, further comprisingdetermining information associated with the high variance region,wherein the information comprises at least one of an x-position of thehigh variance region, a y-position of the high variance region, az-position of the high variance region, an orientation of the highvariance region, or an extent of the high variance region.

AA. A system comprising: one or more processors; and computer-readableinstructions that, when executed by the one or more processors, causethe one or more processors to perform operations comprising: receiving,from a sensor associated with the system, sensor data representing anenvironment; inputting, into a machine learned model, at least a portionof the sensor data; receiving, from the machine learned model, anindication of a low variance region represented in the sensor data;determining an indication of the high variance region based at leastpartly on the indication of the low variance region; determining, basedat least partly on the indication of the low variance region or theindication of the high variance region, that a high variance region ispresent in the environment; and providing the indication of the highvariance region to a component of the system for controlling the system.

AB. The system as paragraph AA recites, wherein the machine learnedmodel is a first machine learned model that outputs a first outputincluding the indication of the low variance region and informationassociated with the low variance region, wherein the informationcomprises at least one of an x-position of the low variance region, ay-position of the low variance region, a z-position of the low varianceregion, an orientation of the low variance region, or an extent of thelow variance region.

AC. The system as paragraph AB recites, wherein determining theindication of the high variance region comprises: inputting the firstoutput into a second machine learned model; receiving, from the secondmachine learned model, the indication of the high variance region,wherein the second machine learned model is trained to detect highvariance regions in image data.

AD. The system as paragraph AB or AC recites, wherein determining theindication of the high variance region comprises: receiving, based atleast partly on analyzing the sensor data using a second machine learnedmodel, a second output indicating at least one feature of the highvariance region; inputting the first output and the second output into athird machine learned model, wherein the third machine learned model istrained to detect high variance regions in sensor data; and receiving,from the third machine learned model, the indication of the highvariance region.

AE. The system as any of claims AB-AD recites, the operations furthercomprising: receiving, based at least partly on analyzing the sensordata using a second machine learned model, a second output indicating aplurality of classifications of an object identified in the sensor dataand a plurality of confidence scores, an individual confidence scorecorresponding to an individual classification; reducing, based at leastpartly on the first output, a threshold associated with a high varianceregion classification of the plurality of classifications; determiningthat a confidence score associated with the high variance regionclassification meets or exceeds the threshold; and determining that thehigh variance region is present in the environment based at least partlyon determining that the confidence score meets or exceeds the threshold.

AF. The system as any of claims AA-AE recites, wherein the indication ofthe high variance region is a bounding box that is determined based atleast partly on one or more of an x-position of the high varianceregion, a y-position of the high variance region, a z-position of thehigh variance region, an orientation of the high variance region, or anextent of the high variance region.

AG. The system as any of claims AA-AF recites, wherein the sensor datais image data and detecting the low variance region comprises detectingat least one of a front view of a face, a side view of a face, or a backof the low variance region in the image data.

AH. The system as any of claims AA-AG recites, wherein determining theindication of the high variance region comprises: performing astatistical analysis of the indication of the low variance region and anassociated portion of the sensor data that is proximate to the lowvariance region; and determining, based at least partly on thestatistical analysis and the associated portion of the sensor data thatis proximate to the low variance region, a position and extents of theindication of the high variance region, wherein the indication of thehigh variance region is determined based at least partly on theindication of the low variance region.

AI. The system as any of claims AA-AH recites, wherein the system is avehicle and the operations further comprise: providing the indication ofthe high variance region to at least one of a planning component or aprediction component for determining a trajectory along which thevehicle is to travel; and controlling the vehicle based at least partlyon the trajectory.

AJ. One or more computer-readable media that, when executed by one ormore processors, cause the one or more processors to perform operationscomprising: receiving, from a sensor associated with a vehicle, sensordata representing an environment within which the vehicle is located;inputting, into a machine learned model, at least a portion of thesensor data; receiving, from the machine learned model, an indication ofa low variance region represented in the sensor data; determining anindication of the high variance region based at least partly on theindication of the low variance region; determining, based at leastpartly on the indication of the low variance region or the indication ofthe high variance region, that a high variance region is present in theenvironment; and providing the indication of the high variance region toa component of the vehicle for controlling the vehicle.

AK. The one or more computer-readable media as paragraph AJ recites,wherein the machine learned model comprises a first machine learnedmodel and the indication of the low variance region comprises a firstoutput, and wherein determining the indication of the high varianceregion comprises: inputting the first output into a second machinelearned model; receiving, from the second machine learned model, theindication of the high variance region, wherein the second machinelearned model is trained to detect high variance regions in image data.

AL. The one or more computer-readable media as paragraph AJ or AKrecites, wherein the machine learned model comprises a first machinelearned model and the indication of the low variance region comprises afirst output, and wherein determining the indication of the highvariance region comprises: receiving, based at least partly on analyzingthe sensor data using a second machine learned model, a second outputindicating at least one feature of the high variance region; inputtingthe first output and the second output into a third machine learnedmodel, wherein the third machine learned model is trained to detect highvariance regions in sensor data; and receiving, from the third machinelearned model, the indication of the high variance region.

AM. The one or more computer-readable media as any of claims AJ-ALrecites, wherein the machine learned model comprises a first machinelearned model and the indication of the low variance region comprises afirst output, the operations further comprising: receiving, based atleast partly on analyzing the sensor data using a second machine learnedmodel, a second output indicating a plurality of classifications of anobject identified in the sensor data and a plurality of confidencescores, an individual confidence score corresponding to an individualclassification; reducing, based at least partly on the first output, athreshold associated with a high variance region classification of theplurality of classifications; determining that a confidence scoreassociated with the high variance region classification meets or exceedsthe threshold; and determining that the high variance region is presentin the environment based at least partly on determining that theconfidence score meets or exceeds the threshold.

AN. The one or more computer-readable media as any of claims AJ-AMrecites, wherein determining the indication of the high variance regioncomprises: performing a statistical analysis of the indication of thelow variance region and an associated portion of the sensor data that isproximate to the low variance region; and determining, based at leastpartly on the statistical analysis and the associated portion of thesensor data that is proximate to the low variance region, a position andextents of the indication of the high variance region, wherein theindication of the low variance region is associated with a firstbounding box and the indication of the high variance region isassociated with a second bounding box.

AO. A method comprising: receiving, from an image capturing deviceassociated with a vehicle, image data representing an environment;inputting at least a portion of the image data into a machine learnedmodel; determining, by the machine learned model, an indication of a lowvariance region associated with the image data; determining, based atleast partly on the indication of the low variance region, an indicationof a high variance region in the image data represented by a boundingbox; providing the bounding box to at least one of a planning componentor a prediction component of the vehicle for determining a trajectoryalong which the vehicle is to travel; and controlling the vehicle basedat least partly on the trajectory.

AP. The method as paragraph AO recites, wherein determining theindication of the high variance region comprises: performing astatistical analysis of the indication of the low variance region and anassociated portion of the image data that is proximate to the lowvariance region; and determining, based at least partly on thestatistical analysis and the associated portion of the image data thatis proximate to the low variance region, a position and extents of thehigh variance region.

AQ. The method as paragraph AO or AP recites, wherein the machinelearned model comprises a first machine learned model and the indicationof the low variance region comprises a first output, and whereindetermining the indication of the high variance region comprises:inputting the first output into a second machine learned model; andreceiving, from the second machine learned model, the bounding box,wherein the second machine learned model is trained to detect highvariance regions in image data.

AR. The method as any of paragraphs AO-AQ recites, wherein the machinelearned model comprises a first machine learned model and the indicationof the low variance region comprises a first output, and whereindetermining the indication of the high variance region comprises:determining, based at least partly on analyzing the image data using asecond machine learned model, a second output indicating at least onefeature of the high variance region; and inputting the first output andthe second output into a third machine learned model, wherein the thirdmachine learned model is trained to determine high variance regions inimage data.

AS. The method as any of paragraphs AO-AR recites, further comprising:receiving, based at least partly on the image data, a plurality ofclassifications of an object identified in the image data and aplurality of confidence scores, an individual confidence scorecorresponding to an individual classification; reducing, based at leastpartly on the indication of the low variance region, a thresholdassociated with a classification of the plurality of classifications,wherein the classification corresponds to a high variation region;determining that a confidence score associated with the classificationmeets or exceeds the threshold; and determining the indication of thehigh variance region based at least partly on determining that theconfidence score meets or exceeds the threshold.

AT. A system comprising: one or more processors; and computer-readableinstructions that, when executed by the one or more processors, causethe one or more processors to perform operations comprising: receivingsensor data representing an environment; determining an indication of alow variance region associated with the sensor data; determining anindication of a high variance region associated with the sensor databased at least in part on the indication of the low variance region; andcontrolling the system based on at least one of the sensor data or theindication of the high variance region.

AU. The system as paragraph AT recites, the operations furthercomprising: determining an additional indication of a high varianceregion associated with the sensor data; and controlling the systemfurther based at least in part on the additional indication of the highvariance region.

AV. The system as paragraph AU recites, the operations furthercomprising: determining, based at least in part on the indication of thehigh variance region and the additional indication of the high varianceregion, a combined indication of the high variance region; andcontrolling the system further based at least in part on the combinedindication of the high variance region.

AW. The system as any of paragraphs AT-AV recites, wherein theindication of the low variance region is one or more of represented inthe sensor data or derived from the sensor data.

AX. The system as any of paragraphs AT-AW recites, wherein: thedetermining the indication of the low variance region is based at leastin part on analyzing at least a portion of the sensor data using aneural network, the low variance region comprises a representation of aface in the sensor data, and the high variance region comprises arepresentation of a pedestrian in the sensor data.

AY. The system as paragraph AX recites, wherein the neural network istrained based at least in part on utilizing another neural network toproject features associated with input data into an image space togenerate reconstructed input data and enforcing consistency between theinput data and the reconstructed input data using a loss function.

AZ. The system as any of paragraphs AT-AY recites, wherein determiningthe indication of the high variance region comprises: inputting theindication of a low variance region into a portion of a machine learnedmodel trained to detect high variance regions in sensor data; andanalyzing the indication of the low variance region by the portion ofthe machine learned model.

BA. The system as any of paragraphs AT-AZ recites, the operationsfurther comprising: receiving a plurality of classifications of anobject identified in the sensor data and a plurality of confidencescores, an individual confidence score corresponding to an individualclassification; reducing a threshold associated with a classification ofthe plurality of classifications associated with the high varianceregion; determining that a confidence score associated with theclassification meets or exceeds the threshold; and determining theindication of the high variance region based at least partly ondetermining that the confidence score meets or exceeds the threshold.

BB. The system as paragraph BA recites, wherein reducing the thresholdassociated with the classification is based at least in part ondetermining the indication of the low variance region.

BC. The system as any of paragraphs AT-BB recites, wherein theindication of the high variance region comprises a bounding box.

BD. One or more computer-readable media that, when executed by one ormore processors, cause the one or more processors to perform operationscomprising: receiving, from a sensor associated with a vehicle, sensordata representing an environment within which the vehicle is located;inputting, into a machine learned model, at least a portion of thesensor data; determining, by the machine learned model, an indication ofa low variance region associated with the sensor data; and determiningan indication of a high variance region based at least in part on aportion of the sensor data associated with the low variance region.

BE. The one or more computer-readable media as paragraph BD recites, theoperations further comprising determining the indication of the highvariance region based at least in part on the indication of the lowvariance region associated with the portion of the sensor data.

BF. The one or more computer-readable media as paragraph BD or BErecites, the operations further comprising determining the indication ofthe high variance region based at least in part on analyzing at leastthe portion of the sensor data using at least one of another machinelearned model or another portion of the machine learned model.

BG. The one or more computer-readable media as any of paragraphs BD-BFrecites, the operations further comprising: providing the indication ofthe high variance region to at least one of a planning component or aprediction component for determining a trajectory along which thevehicle is to travel; and controlling the vehicle based at least partlyon the trajectory.

BH. The one or more computer-readable media as any of paragraphs BD-BGrecites, wherein: the sensor data is image data; the low variance regionis associated with at least one of a front view of a face, a side viewof a face, or a back of a face in the image data; and the high varianceregion is associated with a pedestrian.

BI. A method comprising: receiving data associated with environments ofvehicles; determining annotated data based at least in part on the data,wherein the annotated data comprises an annotated high variance regionin the data and an annotated low variance region in the data; inputtingthe data into a model; determining, by the model, an output comprising ahigh variance output and a low variance output; determining a differencebetween the output and the annotated data; altering parameters of themodel based at least in part on the difference; and transmitting themodel to a vehicle configured to be controlled by another output of themodel.

BJ. The method as paragraph BI recites, wherein the annotated lowvariance region is determined from one or more statistical models.

BK. The method as paragraph BI or BJ recites, wherein the low varianceregion is determined based at least in part on a feature associated withthe data.

BL. The method as paragraph BK recites, the method further comprising:inputting the feature into an additional model; receiving, from theadditional model, a reconstructed output; and determining a loss basedon a difference between the reconstructed output and the data, whereinaltering the one or more parameters is further based at least in part onthe loss.

BM. The method as any of paragraphs BI-BL recites, wherein: the model isa neural network, and the high variance output is based on the lowvariance output, the method further comprising determining an additionalhigh variance output, and further wherein altering the one or moreparameters comprises training the model end-to-end based at least inpart on the low variance output, the high variance output, and theadditional high variance output.

BN. The method as any of paragraphs BI-BM recites, wherein the lowvariance output comprises head detection and the high variance outputcomprises a pedestrian detection.

BO. The method as any of paragraphs BI-BN recites, wherein the datacomprises image data, a batch of image data, or an image space.

BP. A system comprising: one or more processors; and one or morenon-transitory computer-readable media that, when executed by the one ormore processors, cause the system to perform operations comprising:receiving sensor data from a sensor associated with environments ofvehicles; annotating, as annotated data and based on the sensor data, anannotated low variance region associated with the sensor data; traininga model based at least in part on the annotated data and the sensor datato generate a trained model, the trained model configured to output anindication of a low variance region and an indication of a high varianceregion based at least in part on an input; and transmitting the trainedmodel to a vehicle configured to be controlled by another output of themodel.

BQ. The system as paragraph BP recites, wherein the sensor datacomprises at least one of image data, a batch of image data, or an imagespace.

BR. The system as paragraph BQ recites, wherein the annotated lowvariance region is associated with a feature of the sensor datadetermined by the model.

BS. The system as paragraph BR recites, the operations furthercomprising: mapping the feature to a reconstructed input; anddetermining, as a loss, a difference between the sensor data and thereconstructed input, wherein training the model is further based atleast in part on the loss.

BT. The system as paragraph BS recites, wherein mapping the feature toreconstructed input comprises: inputting the feature into an additionalmodel; and receiving, from the additional model, the reconstructedinput.

BU. The system as paragraph BS or BT recites, wherein the indication ofthe high variance region is based at least in part on the indication ofthe low variance region, and wherein training the model comprisestraining the model from end to end.

BV. The system as any of paragraphs BP-BU recites, wherein the annotatedlow variance region is determined from one or more statistical modelsbased at least in part on one or more of entropy, pixel intensity, oraspect ratios associated with indications of low variance regions.

BW. One or more non-transitory computer-readable media that, whenexecuted by one or more processors, cause the one or more processors toperform operations comprising: determining, based on sensor datareceived from a sensor associated with a vehicle, annotated data,wherein the annotated data comprises one or more of an annotated lowvariance region or an annotated high variance region; inputting thesensor data into a model; determining, by the model, an outputcomprising a high variance output and a low variance output; determininga difference between the annotated data and the output; altering one ormore parameters associated with the model based at least in part on thedifference; and transmitting the model to a vehicle configured to becontrolled by another output of the model.

BX. The one or more non-transitory computer-readable media as paragraphBW recites, wherein determining the annotated data comprises determiningthe annotated low variance region based at least in part on astatistical model associated with one or more of the sensor data or anintermediary output of the model based at least in part on the sensordata.

BY. The one or more non-transitory computer-readable media as paragraphBW or BX recites, the operations further comprising: inputting at leasta portion of the sensor data into the model; receiving, as a set offeatures, an intermediate output of the model; inputting the set offeatures into one or more of an additional model or a portion of themodel; receiving, from the one or more of additional model or portion ofthe model, a reconstructed input; and determining a second differencebetween the reconstructed output and the portion of the sensor data,wherein determining the annotated data comprises determining, using astatistical model, the low variance region associated with the set offeatures.

BZ. The one or more non-transitory computer-readable media as paragraphBY recites, wherein altering the one or more parameters is further basedat least in part on the second difference.

CA. The one or more non-transitory computer-readable media as paragraphBY or BZ recites, wherein: the model is a neural network, and the highvariance output is based on the low variance output, the operationsfurther comprising determining an additional high variance output, andfurther wherein altering the one or more parameters comprises trainingthe model end-to-end based at least in part on the low variance output,the high variance output, and the additional high variance output.

CB. The one or more non-transitory computer-readable media as any ofparagraphs BX-CA recites, wherein the low variance output comprises headdetection and the high variance output comprises a pedestrian detection.While the example clauses described above are described with respect toone particular implementation, it should be understood that, in thecontext of this document, the content of the example clauses can also beimplemented via a method, device, system, a computer-readable medium,and/or another implementation. Additionally, any of examples A-CB may beimplemented alone or in combination with any other one or more of theexamples A-CB.

CONCLUSION

While one or more examples of the techniques described herein have beendescribed, various alterations, additions, permutations and equivalentsthereof are included within the scope of the techniques describedherein.

In the description of examples, reference is made to the accompanyingdrawings that form a part hereof, which show by way of illustrationspecific examples of the claimed subject matter. It is to be understoodthat other examples can be used and that changes or alterations, such asstructural changes, can be made. Such examples, changes or alterationsare not necessarily departures from the scope with respect to theintended claimed subject matter. While the steps herein can be presentedin a certain order, in some cases the ordering can be changed so thatcertain inputs are provided at different times or in a different orderwithout changing the function of the systems and methods described. Thedisclosed procedures could also be executed in different orders.Additionally, various computations that are herein need not be performedin the order disclosed, and other examples using alternative orderingsof the computations could be readily implemented. In addition to beingreordered, the computations could also be decomposed intosub-computations with the same results.

What is claimed is:
 1. A method comprising: receiving, from an imagecapturing device associated with a vehicle, image data representing anenvironment; inputting at least a portion of the image data into amachine learned model; determining, by the machine learned model, anindication of a low variance region associated with the image data;determining, based at least partly on the indication of the low varianceregion, an indication of a high variance region in the image datarepresented by a bounding box; providing the bounding box to at leastone of a planning component or a prediction component of the vehicle fordetermining a trajectory along which the vehicle is to travel; andcontrolling the vehicle based at least partly on the trajectory.
 2. Themethod as claim 1 recites, wherein determining the indication of thehigh variance region comprises: performing a statistical analysis of theindication of the low variance region and an associated portion of theimage data that is proximate to the low variance region; anddetermining, based at least partly on the statistical analysis and theassociated portion of the image data that is proximate to the lowvariance region, a position and extents of the high variance region. 3.The method as claim 1 recites, wherein the machine learned modelcomprises a first machine learned model and the indication of the lowvariance region comprises a first output, and wherein determining theindication of the high variance region comprises: inputting the firstoutput into a second machine learned model; and receiving, from thesecond machine learned model, the bounding box, wherein the secondmachine learned model is trained to detect high variance regions inimage data.
 4. The method as claim 1 recites, wherein the machinelearned model comprises a first machine learned model and the indicationof the low variance region comprises a first output, and whereindetermining the indication of the high variance region comprises:determining, based at least partly on analyzing the image data using asecond machine learned model, a second output indicating at least onefeature of the high variance region; and inputting the first output andthe second output into a third machine learned model, wherein the thirdmachine learned model is trained to determine high variance regions inimage data.
 5. The method as claim 1 recites, further comprising:receiving, based at least partly on the image data, a plurality ofclassifications of an object identified in the image data and aplurality of confidence scores, an individual confidence scorecorresponding to an individual classification; reducing, based at leastpartly on the indication of the low variance region, a thresholdassociated with a classification of the plurality of classifications,wherein the classification corresponds to a high variation region;determining that a confidence score associated with the classificationmeets or exceeds the threshold; and determining the indication of thehigh variance region based at least partly on determining that theconfidence score meets or exceeds the threshold.
 6. A system comprising:one or more processors; and computer-readable instructions that, whenexecuted by the one or more processors, cause the one or more processorsto perform operations comprising: receiving sensor data representing anenvironment; determining an indication of a low variance regionassociated with the sensor data; determining an indication of a highvariance region associated with the sensor data based at least in parton the indication of the low variance region; and controlling the systembased on at least one of the sensor data or the indication of the highvariance region.
 7. The system as claim 6 recites, the operationsfurther comprising: determining an additional indication of a highvariance region associated with the sensor data; and controlling thesystem further based at least in part on the additional indication ofthe high variance region.
 8. The system as claim 7 recites, theoperations further comprising: determining, based at least in part onthe indication of the high variance region and the additional indicationof the high variance region, a combined indication of the high varianceregion; and controlling the system further based at least in part on thecombined indication of the high variance region.
 9. The system as claim6 recites, wherein the indication of the low variance region is one ormore of represented in the sensor data or derived from the sensor data.10. The system as claim 6 recites, wherein: the determining theindication of the low variance region is based at least in part onanalyzing at least a portion of the sensor data using a neural network,the low variance region comprises a representation of a face in thesensor data, and the high variance region comprises a representation ofa pedestrian in the sensor data.
 11. The system as claim 10 recites,wherein the neural network is trained based at least in part onutilizing another neural network to project features associated withinput data into an image space to generate reconstructed input data andenforcing consistency between the input data and the reconstructed inputdata using a loss function.
 12. The system as claim 6 recites, whereindetermining the indication of the high variance region comprises:inputting the indication of a low variance region into a portion of amachine learned model trained to detect high variance regions in sensordata; and analyzing the indication of the low variance region by theportion of the machine learned model.
 13. The system as claim 6 recites,the operations further comprising: receiving a plurality ofclassifications of an object identified in the sensor data and aplurality of confidence scores, an individual confidence scorecorresponding to an individual classification; reducing a thresholdassociated with a classification of the plurality of classificationsassociated with the high variance region; determining that a confidencescore associated with the classification meets or exceeds the threshold;and determining the indication of the high variance region based atleast partly on determining that the confidence score meets or exceedsthe threshold.
 14. The system as claim 13 recites, wherein reducing thethreshold associated with the classification is based at least in parton determining the indication of the low variance region.
 15. The systemas claim 6 recites, wherein the indication of the high variance regioncomprises a bounding box.
 16. One or more computer-readable media that,when executed by one or more processors, cause the one or moreprocessors to perform operations comprising: receiving, from a sensorassociated with a vehicle, sensor data representing an environmentwithin which the vehicle is located; inputting, into a machine learnedmodel, at least a portion of the sensor data; determining, by themachine learned model, an indication of a low variance region associatedwith the sensor data; and determining an indication of a high varianceregion based at least in part on a portion of the sensor data associatedwith the low variance region.
 17. The one or more computer-readablemedia as claim 16 recites, the operations further comprising determiningthe indication of the high variance region based at least in part on theindication of the low variance region associated with the portion of thesensor data.
 18. The one or more computer-readable media as claim 16recites, the operations further comprising determining the indication ofthe high variance region based at least in part on analyzing at leastthe portion of the sensor data using at least one of another machinelearned model or another portion of the machine learned model.
 19. Theone or more computer-readable media as claim 16 recites, the operationsfurther comprising: providing the indication of the high variance regionto at least one of a planning component or a prediction component fordetermining a trajectory along which the vehicle is to travel; andcontrolling the vehicle based at least partly on the trajectory.
 20. Theone or more computer-readable media as claim 16 recites, wherein: thesensor data is image data; the low variance region is associated with atleast one of a front view of a face, a side view of a face, or a back ofa face in the image data; and the high variance region is associatedwith a pedestrian.